| Author |
 Topic  |
|
|
Rob
USA
2615 Posts |
 Posted - 02/28/2013 : 13:42:14
|
It would seem the time has come to begin investigating the options for supporting touch screens. To be clear, I'm only referring to regular Windows 8, not Windows RT (the stripped down, Android-like, low-powered version); this would only be for systems running an x86/x64 CPU.
So, I ask you all: How would you like to see S+ operate in a touch environment?
How would you see setting the "stroke" capture?
2 or 3 fingers down, moving together? (could conflict with a 2-finger scroll touch gesture)
A stationary finger with another one drawing? (imagine your thumb is down and stays in one place, and you draw with your index finger, could conflict with a pinch-zoom touch gesture)
Keep in mind any existing/expected touch gestures, like 2-finger scrolling, for example. If we captured 2-fingers down as a gesture, 2-finger scrolling would be overridden. Just want to make sure I approach this the right way.
It may be possible to have an ignore gesture using a nearly stationary finger. For example, let's say we use 2-finger drawings as a S+ capture, which overrides a normal 2-finger scroll gesture (not S+ gesture). Perhaps if you have a stationary finger (thumb down) and draw a 2-finger gesture (where they're moving side by side, together), S+ forwards only the 2-finger touch input to the window. So S+ consumes the thumb input (so Windows/the app doesn't think the thumb is down) and sends the 2-finger touch input on, so Windows/the app thinks you're only doing a 2-finger scroll (for example). It could be tricky, meaning you'd have to have your thumb and 2 fingers touch down almost at the same time (so S+ doesn't have to interfere with normal taps), but I think it's something that could be reasonable and S+ users able to easily adjust to hitting the ignore override the right way; it would only be for overriding the capture gesture, so it shouldn't get in the way much.
Initially, I envision only tracking a single touch point for gesture recognition, to fit into the existing S+ functionality. Meaning, assuming a 2-finger gesture capture, only the first finger's movements would be captured for recognition. Not that it would matter much for 2-finger capture as they'd be drawing essentially the same thing, but just to make sure I'm clear on how the initial release of this functionality would be. To support multi-finger gestures drawing different paths would require an entire S+ rewrite...which is something that still needs to happen, but adding simple touch capabilities is something I'd like to add soon, if possible.
Technically, S+ could capture only on hover (draw a gesture without touching the screen), but I imagine this may be problematic and inconsistent (accidentally bumping the screen, low-quality hardware not picking up a finger until it's just about touching, etc.)
Note that I would be dynamically binding to the Windows 8 touch API calls, so the same S+ EXE/DLL would run fine all the way back to XP, just disabling the option for touch in anything less than Windows 8. I'm aware Windows Vista/7 supports touch, but that entire API has been deprecated and replaced in Windows 8, so I really don't want to have to duplicate the effort for this considering that going forward, Windows will use the newer API. Additionally, the Windows Vista/7 touch API is "greedy", meaning, whichever touch window first receives the input will get all touch input until it loses focus. Where Windows 8 allows S+ to register itself as the touch input target window, so all touch goes to S+ first, then if it's not recognized as a gesture capture, S+ forwards the touch event on to the appropriate window as the user intended.
Since I don't actually have a touchscreen system, development may be problematic! 
I will only be able to use the Windows Simulator, which should cover it, but only up to a point for a Beta release...then I'll need to buy something.
Let me know your thoughts, ideas, potential challenges, etc. |
|
|
breakcore
Russia
74 Posts |
Posted - 02/28/2013 : 18:22:00
|
We need touch screens that can differentiate the pressing fingers (and other parts lol, like nails, knuckles...)). This would definitely reduce a "stroke capture" thoughts.
S+ for touch screen interface will probably have even bigger potential than it has now.
Had no chance to use tablet pc. To be honest, I don't think I will ever be able to fully work on pc without a mouse. Unless it become somehow integrated in the hand, so that fingers will produce a mouse clicks when touching any surface  |
 |
|
|
Rob
USA
2615 Posts |
Posted - 02/28/2013 : 19:04:56
|
I'm with you...a mouse is a necessity for me. But I still think it would be cool for S+ to support touch screens anyway, plus it's a new challenge
I just finished putting together the code to capture all touch input and print out which WM_POINTER* message was received to the debugger. Too bad that was the easy part.
To build this functionality, S+ will have to take in and keep track of each finger contact message (and history for replay and if the capture sequence wasn't matched within the time frame), and not just forward the message along, but manually create and populate each contact point struct and inject all of the details about the touch, including the WM_POINTERUPDATE messages which are constantly coming through when a finger is on or anywhere near the surface.
I reached a big milestone by having S+ able to capture all touch (and pen) input, but the rest is going to be a big pain! |
|
|
|
margas
Italy
14 Posts |
Posted - 03/03/2013 : 12:26:02
|
Mh, it's very difficult. both 1 or 2 fingers conflict with windows gestures. what about activating input tapping with 3 fingers, then drawing with 1?
Or a stationary finger with 2 drawing |
|
|
|
Rob
USA
2615 Posts |
Posted - 03/03/2013 : 15:07:19
|
quote:
Originally posted by margasOr a stationary finger with 2 drawing
I'm kind of thinking that may be the best option. It seems like it would avoid common gestures for Windows or other apps, and feels like it would be pretty comfortable. I could see potentially needing a (user defined) speed option, like with the TouchPad support to allow you to increase the scale/sensitivity of the drawing since having a stationary finger (like thumb) would limit how far your two fingers that are drawing can go. So you would start a S+ gesture, and (if you set the increased sensitivity) the line drawn on the screen wouldn't draw right under your finger, but be larger scale so you can easily see what gesture you're drawing. Heck, maybe for touch screen have a pre-defined area of the screen where the gesture line is always drawn. |
|
|
|
margas
Italy
14 Posts |
Posted - 03/03/2013 : 15:33:02
|
quote:
Originally posted by Rob
Heck, maybe for touch screen have a pre-defined area of the screen where the gesture line is always drawn.
yes, this could be a solution |
|
|
|
n/a
1 Posts |
Posted - 06/25/2013 : 16:06:08
|
As a fellow developer I find it's easiest to start very simple and iterate from there. Ship and see how people like it, how you like it, what's easy what's not, and determine next steps.
So, I don't know what's easiest for you but I'll guess that the simplest way given the existing interface is to offer 2-finger, 3-finger, and 4-finger options in addition to Right-Click etc in the current UI. The rest of the application would continue to operate the way it does now, and the only change is you've got to find some way to tie into the OS to capture that kind of input (hopefully that's all nicely wrapped for you in Win8...). The hard part may be coping with whatever bugs there are in that wrapping.
I'm specifically looking for something that lets me take 3 fingers and swipe left for Back, right for Forward, and maybe make a squiggle for Close Window/Tab (basically just fires a Ctrl+W at the focused app). Nothing complicated, and it'd make my day to day usage easier. |
|
|
|
elerouxx
Brazil
40 Posts |
Posted - 08/23/2013 : 18:55:38
|
Hey Rob!
Sorry, I've been so busy at the conservatory. But I still use S+ every day, making music! my colleagues are always impressed. (remember 'making music'?)
I'm arriving late at the discussion but, I wanted to know, how is your progress with touch screen devices?
It's about time I change my old tabletPC, which has no touch capabilities, just a wacom pen, and has windows 7 installed. The newer options are windows 8 tablets with or without the stylus. I wonder if I will be able to use S+ the same way I'm used to in my tabletpc, even without a stylus. (I'm trying to get one with stylus support but they are expensive -he surface pro being the ideal choice so far).
I imagine that, for left mouse button gestures, drawing the gesture with one finger should work just the same. Or not? In my specific case I use the left mouse button, which is equal that just putting the pen against the screen and start drawing around. But I understand left-button gestures have much less applications for general use.
I agree with b9chris, in which the easier way (for the user at least) would be having 2- or 3-finger gestures, besides the single finger gesture, and right-click gesture (which would be to hold the finger for some seconds, then do the gesture).
Probably most people will end up using 3-finger gestures, which interferes less with left and right clicking, and with zoom/rotate gestures which use 2 fingers usually.
|
|
|
|
Rob
USA
2615 Posts |
Posted - 08/23/2013 : 19:47:41
|
Of course I remember you
There honestly hasn't been much news on that front. I don't have a touch device, so that certainly presents a challenge! It's also a little more complicated since there can only be one window which is designated to receive all touch input, which means S+ would also be responsible for ensuring that every non-S+ touch event is forwarded to the proper window in the proper manner, etc.
Interestingly, Leap Motion reached out to me to see if they could be of any assistance in integrating the Leap functionality into S+. Which is certainly something I'm interested in pursuing along with touch (the Leap integration would be similar to touch in the sense of probably starting with a 3-finger gesture setup); however, until I actually own a touch device, it's simply not going to happen (as it cannot, in practical terms). I have been eyeing the Surface Pro...maybe I'll eventually bite the bullet |
|
|
|
elerouxx
Brazil
40 Posts |
Posted - 08/24/2013 : 06:53:19
|
quote:
Originally posted by Rob
I have been eyeing the Surface Pro...maybe I'll eventually bite the bullet
Same here. A friend of mine is in NY right now, and I have the opportunity to buy either a Surface Pro or a cheaper and lighter device, but the only choice I have in the later option is an Asus Vivotab without a digitizer ($449 at bestbuy, great comments about it).
Here in Brazil, I could get a DELL LATITUDE 10 tablet (with digitizer and atom CPU) which also had great opinions, but that would be more expensive than a surface pro bought in the U.S. (ouch!). Crazy world, here people pays the same for a phone.
I'm afraid the Surface will be brutally taxed when it arrives because it exceeds the legal price for personal imports at customs here. On the other hand I wonder if I could survive with a touch only device after all these years, and also if S+ works at all on that.
All I can tell you is that tablet PCs have always been great, despite the reduced market they got, and I'm pretty sure you will enjoy one of these new devices which, at the weight and cost of an iPad, deliver a full computer. I wouldn't think twice if I lived in the u.s. :)
About Leap Motion, I think it's a great project. And Strokes Plus is by far the best gesture application around - mac users envy it, specially when I show them my music-writing environment. I'm sure good things will come out from both Leap Motion and S+ together! |
|
|
|
elerouxx
Brazil
40 Posts |
Posted - 08/24/2013 : 21:31:07
|
Rob, it just occurred to me that the easiest solution may be under ur noses. Two and three fingers would be nice, but the '1 finger hold delay' for synaptics touchpad works very well!
I think you might try to implement 'hold delay' as an option not only for synaptics but for the whole interface! in theory, it would work right away with tablets - the user would hold his finger for, say, 300 ms, and only then drag it to draw the gesture. If the user holds his finger long enough (500 ms for instance) he would activate the Right mouse button instead, as configured in all windows tablets, table pcs or using graphic tablets like wacom.
Also, I think mouse users would also benefit from this. It feels very natural pressing the left mouse button and 1/4 sec later draw a gesture. Nobody clicks the left mouse button and holds still without a purpose, except when airbrushing something in a paint application.
I only don't know the potential conflicts with the press-and-hold for right button feature, at system level (mean, if you can check for holding delay without interfering with the longer delay read by the OS to be interpreted as right click).
Is this a good idea or I'm high on camomilla tea?
|
|
|
|
Rob
USA
2615 Posts |
Posted - 08/25/2013 : 12:45:03
|
You're high on camomilla tea
It's not the implementation of capturing/recognition that is the complicated part, it's that S+ has to register itself as the window which receives all touch input (at all times), which means that I have to recreate and relay all touch interactions to the intended (and correct) recipient of the touch input, if it wasn't recognized as a gesture...and do it all without interfering with the overall touch experience for the user.
Once the OS captures the touch and converts it to a right-mouse button event, then S+ works as it does today; but to intercept the touch input initially, S+ would also have the responsibility of delivering all touch events/data properly and perfectly all of the time, for all applications. That's where I feel it's not going to be as easy as it may seem.
This may or may not be a hard thing to do, but without an actual touch device, it's all speculation. |
|
|
|
Rob
USA
2615 Posts |
Posted - 08/25/2013 : 12:46:57
|
| However, if a standard single finger touch is a left button event, then yes, it would just be a matter of adding the Synaptics logic into the existing code and then you're not actually high on anything |
|
|
|
elerouxx
Brazil
40 Posts |
Posted - 08/25/2013 : 23:07:05
|
quote:
Originally posted by Rob
However, if a standard single finger touch is a left button event, then yes, it would just be a matter of adding the Synaptics logic into the existing code and then you're not actually high on anything
Actually I was thinking exactly about adding the same synaptics 'hold delay' logic as an option to the mouse LEFT button event, because -in theory- single finger touch is the same as the mouse left button.
Still, if it's not too complicated, maybe you could give it a try, as an additional option besides the 'cancel delay' box (not only for LEFT but for any other button).
Actually, it would be the opposite to 'cancel delay' - instead of overriding S+ by holding still, the user would override S+ by moving the cursor immediately. Thus, holding still for a fraction of a second would allow him to draw the gesture. I think it could be an interesting addition even for mouse or graphic pen gestures.
I am positive but cannot be sure about how it will work on touch devices. My friend who took a trip to NY will try to buy Asus vivotab smart, $450 at bestbuy. If everything is ok and I get my tablet tomorrow, I promise I'll try to help you in any way I can.
|
|
|
|
elerouxx
Brazil
40 Posts |
Posted - 08/26/2013 : 22:21:45
|
and... the toy arrived! I really love it. I definitely recommend this kind of devices, even on the cheapest side like mine. I have to admit the guys at MS did an awesome work with windows 8 and the touch interface.
And I think S+ gives it the functionallity it misses, because fingers don't have buttons or modifier keys... so some gestures help a lot. I'll try to create a couple of them now, specific for the tablet.
Now I can assure you that just touching the screen is the same as left-clicking and dragging, because I can draw the S+ gesture this way. So I think this method of 'hold delay' should work very well. |
|
|
|
elerouxx
Brazil
40 Posts |
Posted - 09/03/2013 : 20:13:40
|
... However, some applications work in different ways.
When there are vertical scroll bars, in some applications dragging a finger means trying to scroll down the document. IN these cases I can only start drawing sidewards, not up and down.
Metro interface applications love fingers, but the pen definitely adds a lot of functionality to a windows tablet, when using desktop apps.
|
|
|
|
Joerrk
4 Posts |
Posted - 09/04/2013 : 09:38:21
|
I just read elerouxx sugestion.
At first I thougt, that this doubles the Windows standard behaviour, but there are 2 advantages
1) I can even use it with the mouse left buttion
2) I have set hold delay to 110ms, thisis much shorter than the shortest Windows7 setting for right touch click
I would volenteer to test ist on a convertable Laptop.
BTW, convertables like the packard bell butterfly touch are quit cheap on ebay, and with like 9h Battery run time and a core2duo processor pretty good value. It is equivalent to the acer 1825
http://www.ebay.de/dsc/Notebooks-Netbooks-/175672/i.html?_ipg=&LH_TitleDesc=1&_from=&LH_Complete=1&_armrs=1&_nkw=butterfly+touch
|
|
|
|
Rob
USA
2615 Posts |
Posted - 09/24/2013 : 08:52:33
|
Just a short follow up. I briefly looked into the click and hold option, but it's far more complicated than the Synaptics logic due to the way the Synaptic information is fed into the function. Also, however, the TouchPad code gets to be in its own sandbox, which makes a big difference.
For example, touch and hold on a TouchPad is something completely outside of Windows, meaning only touching (not tapping) is not an event which Windows is aware of. So S+ monitoring or intercepting it doesn't interfere with anything the user is trying to do.
For a mouse event, it's a whole different ball game. A click is supposed to do something, so intercepting the click and monitoring for movement, time, etc. greatly complicates matters and is a challenge to make sure S+ wouldn't cause any noticeable lag, delay, or interference at all with any app. Non-primary mouse buttons would be less of an issue, as it is with right-click; but seamlessly intercepting the left-click needs more finesse.
It certainly seems possible, but is not something which is quite as quick to implement as it may have seemed initially. |
|
|
|
Albert
2 Posts |
Posted - 02/03/2014 : 07:59:40
|
Hello,
how is your progress with touchscreen devices?
I am using a surface pro and I would love to test it - would it be possible to use it also with the surface digitizer? |
|
|
|
Rob
USA
2615 Posts |
Posted - 02/03/2014 : 08:59:46
|
Sicne I do not have a touch device, progress hasn't really progressed...at all =)
I plan(ned) to get a Surface Pro 2, but I can't really afford that luxury at the moment. |
|
|
|
xpectmore
14 Posts |
Posted - 05/30/2014 : 11:08:31
|
i suggest to rewrite all strokes to touches screens and to set an ability to on/off on stroke by touching upper or bottom screen zone....
an ideea is on arrow up like stroke then put on S+ ,obviously on stroke "arrow down" then off...
i think to rewrite all my strokes to minimal movements that works to all system operation where s+ works...
and combine movements with a .exe server that support gui writed maybe in freebasic x86(x64 doesn't exist) |
Edited by - xpectmore on 05/30/2014 11:10:35 |
|
|
|
Strokeman
33 Posts |
Posted - 06/13/2014 : 13:24:00
|
quote:
Originally posted by Rob
I plan(ned) to get a Surface Pro 2, but I can't really afford that luxury at the moment.
yea that thing is yummy but bit costly. me planning the same as i am heavy Pc user & now traveling more but tabs/phones running ishit/dumdroid/etc doesn't really come close to that experience. Surface pro (barring small display(con of something portable)) is superb.
surface pro (1st version) is now cheap though
Overall IMO, gestures for touchscreen would be trickier to implement &
most importantly carry less productivity RELATIVE TO using S+ with nontouch screen device using mouse/joystick/etc. Fact remains that with touch screen devices, mouse, KB became extinct for a reason. The "what-u-see-is-what-u-touch" principle came into picture & given that most touch screens are still not too larger than a 2 * stretched hand, need for S+ like stuff would only be users who are speed freaks ...combining multiple touch actions into one, etc.
As some1 said above, for touch S+, Always on S+ wud be a problem/conflicting. Start/Stop of S+ using some touch gesture wud be better. (On android,remember using a web browser named dolphin which has gesture control)
Regards
|
|
|
|
AlmaChua
1 Posts |
Posted - 01/10/2017 : 10:16:47
|
How far has your Plan grown? Please excuse my bad english first - i'm German.
My wish for StrokesPlus would be a support WITHOUT a button held down (i.e. right MouseButton), because simple Tablets with capacitive Displays have not a button at the pen. There are several capacitive Windows-Tablets on the market, which are much cheaper than the Surface. On the other hand, all functions on these tablets come without a button and would be disturbed by StrokesPlus.
Maybe it is a good idea to place a free position-able button on the screen. This button could be pressed with the thumb of that hand, which helds the tablet. So the user can choose between using StrokesPlus or the rest of functions. I belive, an own area for StrokesPlus only would stand in the users way. |
Edited by - AlmaChua on 01/12/2017 05:52:45 |
|
|
| |
Topic |
|
|
|
[PLANNED] Touch Screen Support
Forum Locked
