最新消息:雨落星辰是一个专注网站SEO优化、网站SEO诊断、搜索引擎研究、网络营销推广、网站策划运营及站长类的自媒体原创博客

Extract or crop specific part of image using Bounds obtained from accessibility node info in android java - Stack Overflow

programmeradmin2浏览0评论

I am developing an android app to automate data entry and captcha filling on another android app. I have already automated form filling using Android accessibility service, the only part is left is to decode captcha text from the captcha image on screen.

To decode captcha I am following below steps:

  1. With the help of android Media projection, Take the full screenshot of the mobile screen where captcha is present.

  2. Crop the captured screenshot by creating another Bitmap from the image captured on step 1 with Bonds(rect.left,rect.top,rect.width,rect.height) obtained from accessibility node of captcha element.

  3. Pass the cropped image for decoding captcha using image to text extraction(OCR).

The problem is that when i am creating cropped image with Bonds(rect.left,rect.top,rect.width,rect.height) obtained from accessibility node, I am unable to capture full captcha part.

To get captcha part of image I am creating another image using Bonds(rect.left,rect.top,rect.width,rect.height) obtained from accessibility node but its not capturing required part of image.

Locate the element and got Bounds of the captcha element by below code from accessibilityNodeInfo (MyAccessibilityService.java)

List<AccessibilityNodeInfo> findAccessibilityNodeInfosCaptchas = accessibilityNodeInfo.findAccessibilityNodeInfosByViewId("app.package.name:id/captcha");
            if (findAccessibilityNodeInfosCaptchas.size() > 0) {

                Rect rect = new Rect(0,0,0,0);
                findAccessibilityNodeInfosCaptchas.get(0).getBoundsInScreen(rect);
                Log.d(strTag,"Rect"+rect);

                ScreenCaptureService.rectleft = rect.left;
                ScreenCaptureService.rectTop = rect.top;
                ScreenCaptureService.rectWidth = rect.width();
                ScreenCaptureService.rectHeight = rect.height();

            }

Capture the full screenshot of scree using Media Projection

Screenshot of mobile screen

cropped image

Code for screenshot and crop image below(ScreenCaptureService.java) Screen Size 1080X2285 Rect(162, 992 - 502, 1090)

public void onImageAvailable(ImageReader reader) {

    FileOutputStream fos = null;
    Bitmap bitmap = null;
    Bitmap bitmapTmp = null;
    stopProjection();
    try (Image image = mImageReader.acquireLatestImage()) {
        if (image != null) {
            Image.Plane[] planes = image.getPlanes();
            ByteBuffer buffer = planes[0].getBuffer();
            int pixelStride = planes[0].getPixelStride();
            int rowStride = planes[0].getRowStride();
            int rowPadding = rowStride - pixelStride * mWidth;
            Log.d("DebugApp", "onImageAvailable");
            // create bitmap
            bitmap = Bitmap.createBitmap(mWidth + rowPadding / pixelStride, mHeight, Bitmap.Config.ARGB_8888);
            bitmap.copyPixelsFromBuffer(buffer);

            // write bitmap to a file
            fos = new FileOutputStream(mStoreDir + "/myscreen_" + IMAGES_PRODUCED + ".png");
            bitmappress(Bitmap.CompressFormat.JPEG, 100, fos);

            /// create cropped image of captcha page

            //bitmapTmp = Bitmap.createBitmap(bitmap,162,992,502,1090);
            Log.d("DebugApp", "left top width height " + ScreenCaptureService.rectleft + ":" + ScreenCaptureService.rectTop + ":" + ScreenCaptureService.rectWidth + ":" + ScreenCaptureService.rectHeight);
            bitmapTmp = Bitmap.createBitmap(bitmap,
                ScreenCaptureService.rectleft,
                ScreenCaptureService.rectTop,
                ScreenCaptureService.rectWidth,
                ScreenCaptureService.rectHeight);
            // write bitmap to a file
            fos = new FileOutputStream(mStoreDir + "/myscreen_tmp" + IMAGES_PRODUCED + ".png");
            bitmapTmppress(Bitmap.CompressFormat.JPEG, 100, fos);

            ///

            IMAGES_PRODUCED++;
            Log.e("DebugApp", "captured image: " + IMAGES_PRODUCED);
            //stopProjection();
        }

    } catch (Exception e) {
        e.printStackTrace();
        Log.d("DebugApp", "Exp " + e.toString());

    } finally {
        if (fos != null) {
            try {
                fos.close();
            } catch (IOException ioe) {
                ioe.printStackTrace();
            }
        }

        if (bitmap != null) {
            bitmap.recycle();
        }

    }
}

与本文相关的文章

发布评论

评论列表(0)

  1. 暂无评论