I am developing an android app to automate data entry and captcha filling on another android app. I have already automated form filling using Android accessibility service, the only part is left is to decode captcha text from the captcha image on screen.

To decode captcha I am following below steps:

1. With the help of android Media projection, Take the full screenshot of the mobile screen where captcha is present.

2. Crop the captured screenshot by creating another Bitmap from the image captured on step 1 with Bonds(rect.left,rect.top,rect.width,rect.height) obtained from accessibility node of captcha element.

3. Pass the cropped image for decoding captcha using image to text extraction(OCR).

The problem is that when i am creating cropped image with Bonds(rect.left,rect.top,rect.width,rect.height) obtained from accessibility node, I am unable to capture full captcha part.

To get captcha part of image I am creating another image using Bonds(rect.left,rect.top,rect.width,rect.height) obtained from accessibility node but its not capturing required part of image.

Locate the element and got Bounds of the captcha element by below code from accessibilityNodeInfo (MyAccessibilityService.java)

List<AccessibilityNodeInfo> findAccessibilityNodeInfosCaptchas = accessibilityNodeInfo.findAccessibilityNodeInfosByViewId("app.package.name:id/captcha");
            if (findAccessibilityNodeInfosCaptchas.size() > 0) {

                Rect rect = new Rect(0,0,0,0);
                findAccessibilityNodeInfosCaptchas.get(0).getBoundsInScreen(rect);
                Log.d(strTag,"Rect"+rect);

                ScreenCaptureService.rectleft = rect.left;
                ScreenCaptureService.rectTop = rect.top;
                ScreenCaptureService.rectWidth = rect.width();
                ScreenCaptureService.rectHeight = rect.height();

            }

Capture the full screenshot of scree using Media Projection

Screenshot of mobile screen

cropped image

Code for screenshot and crop image below(ScreenCaptureService.java) Screen Size 1080X2285 Rect(162, 992 - 502, 1090)

public void onImageAvailable(ImageReader reader) {

    FileOutputStream fos = null;
    Bitmap bitmap = null;
    Bitmap bitmapTmp = null;
    stopProjection();
    try (Image image = mImageReader.acquireLatestImage()) {
        if (image != null) {
            Image.Plane[] planes = image.getPlanes();
            ByteBuffer buffer = planes[0].getBuffer();
            int pixelStride = planes[0].getPixelStride();
            int rowStride = planes[0].getRowStride();
            int rowPadding = rowStride - pixelStride * mWidth;
            Log.d("DebugApp", "onImageAvailable");
            // create bitmap
            bitmap = Bitmap.createBitmap(mWidth + rowPadding / pixelStride, mHeight, Bitmap.Config.ARGB_8888);
            bitmap.copyPixelsFromBuffer(buffer);

            // write bitmap to a file
            fos = new FileOutputStream(mStoreDir + "/myscreen_" + IMAGES_PRODUCED + ".png");
            bitmappress(Bitmap.CompressFormat.JPEG, 100, fos);

            /// create cropped image of captcha page

            //bitmapTmp = Bitmap.createBitmap(bitmap,162,992,502,1090);
            Log.d("DebugApp", "left top width height " + ScreenCaptureService.rectleft + ":" + ScreenCaptureService.rectTop + ":" + ScreenCaptureService.rectWidth + ":" + ScreenCaptureService.rectHeight);
            bitmapTmp = Bitmap.createBitmap(bitmap,
                ScreenCaptureService.rectleft,
                ScreenCaptureService.rectTop,
                ScreenCaptureService.rectWidth,
                ScreenCaptureService.rectHeight);
            // write bitmap to a file
            fos = new FileOutputStream(mStoreDir + "/myscreen_tmp" + IMAGES_PRODUCED + ".png");
            bitmapTmppress(Bitmap.CompressFormat.JPEG, 100, fos);

            ///

            IMAGES_PRODUCED++;
            Log.e("DebugApp", "captured image: " + IMAGES_PRODUCED);
            //stopProjection();
        }

    } catch (Exception e) {
        e.printStackTrace();
        Log.d("DebugApp", "Exp " + e.toString());

    } finally {
        if (fos != null) {
            try {
                fos.close();
            } catch (IOException ioe) {
                ioe.printStackTrace();
            }
        }

        if (bitmap != null) {
            bitmap.recycle();
        }

    }
}