How can I reliably simulate touch events on Android without root (like Automate and Tasker)?

Question

How can I reliably simulate touch events on Android (without rooting) from Java outside my app which runs as a background service?

While this question has been asked before, most answers utilise ADB. (such as How to simulate touch events on Android device?)

https://github.com/chetbox/android-mouse-cursor offers a good solution using Accessibility, but is not very reliable as not all views respond to it, and games do not respond at all most of the time.

private void click() {
  AccessibilityNodeInfo nodeInfo = getRootInActiveWindow();
  if (nodeInfo == null) return;

  AccessibilityNodeInfo nearestNodeToMouse = findSmallestNodeAtPoint(nodeInfo, cursorLayout.x, cursorLayout.y + 50);

  if (nearestNodeToMouse != null) {
    logNodeHierachy(nearestNodeToMouse, 0);
    nearestNodeToMouse.performAction(AccessibilityNodeInfo.ACTION_CLICK);
  }

  nodeInfo.recycle();
}

This is the current code used by https://github.com/chetbox/android-mouse-cursor.

Android Version is 8.0, stock Android

Is there a better, more reliable way to simulate these touch events from Java? Thanks in advance!

Solution

As suggested, the best way to simulate touch events since Nougat (API 24) is by using an accessibility service and the AccessibilityService#dispatchGesture method.

Here is how I did to simulate a single tap event.

// (x, y) in screen coordinates
private static GestureDescription createClick(float x, float y) {
    // for a single tap a duration of 1 ms is enough
    final int DURATION = 1;

    Path clickPath = new Path();
    clickPath.moveTo(x, y);
    GestureDescription.StrokeDescription clickStroke =
            new GestureDescription.StrokeDescription(clickPath, 0, DURATION);
    GestureDescription.Builder clickBuilder = new GestureDescription.Builder();
    clickBuilder.addStroke(clickStroke);
    return clickBuilder.build();
}

// callback invoked either when the gesture has been completed or cancelled
callback = new AccessibilityService.GestureResultCallback() {
    @Override
    public void onCompleted(GestureDescription gestureDescription) {
        super.onCompleted(gestureDescription);
        Log.d(TAG, "gesture completed");
    }

    @Override
    public void onCancelled(GestureDescription gestureDescription) {
        super.onCancelled(gestureDescription);
        Log.d(TAG, "gesture cancelled");
    }
};

// accessibilityService: contains a reference to an accessibility service
// callback: can be null if you don't care about gesture termination
boolean result = accessibilityService.dispatchGesture(createClick(x, y), callback, null);
Log.d(TAG, "Gesture dispatched? " + result);

To perform other gestures, you might find useful the code used for testing the AccessibilityService#dispatchGesture implementation.

EDIT: I link a post in my blog with an introduction to Android accessibility services.

Comments

[Akash Karnatak]

is there a way to implement touch on api level lower than 24. I have noticed that your app "eva_facial_mouse" does perform taps on android phone with api level below 24.

[Cesar Mauri]

@AkashKarnatak, it is possible until to some extent. EVA uses the AccessibilityNodeInfo#performAction method to perform taps. This means you first need to find the right node you want to tap on. This usually involves searching a tree of accessibility nodes starting from the root. However, not all apps provide such a tree being the games, in my experience, the major offenders. Here is how I actually implemented in EVA: github.com/cmauri/eva_facial_mouse/blob/…

[matanster]

Can you kindly provide few more words about this API? is it an API open to use by any application, that can simulate any form of touch stroke or tap by freely specifying the geometrical and temporal (touch path and velocity at each point in the path) touch paths up to the point of allowing application developers to actually develop accessibility solutions that replace the need to touch the phone's display area? It might look like that on first look.

[Flyview]

This worked for me thank you! It should be noted the accessibility service xml needs to have "android:canPerformGestures=true" for this to work.

[largoWinch]

The problem that i see with this option is that, if i am not wrong, you have a limit to make the "addStroke" method, specifically about 10 times. If you go to the source code of the GestureDescription.java, you can see a variable calle MAX_STROKE_COUNT set it to 10, and in the "addStroke" method ask about the variable. This is a problem only if you don't want to make a lot of strokes.

[Paul Lam]

do we need to request any permission for this ? if so then how can we request the permission(s) ?

[fatbringer]

would it be possible if i send the (x,y) coordinates from another device through a tcp socket, and pass these to simulate a double tap on the same spot ?

[Cesar Mauri]

@fatbringer it may work provided that the server (the app in the device which is receiving the coordinates) is running in the context of an accessibility server

[fatbringer]

@CesarMauri im not familiar with accessibility servers. Are you referring to like ease of access kind of settings ?

[Cesar Mauri]

I meant to say "accessibility service", my mistake @fatbringer.

[fatbringer]

@CesarMauri aye got it i will go check it out