Request for Enhanced Eye Region Segmentation in Single-Eye Videos

[zxie52]

MediaPipe Solution (you are using)

MediaPipe Iris

Programming language

Python

Are you willing to contribute it

Yes

Describe the feature and the current behaviour/state

Enhanced Eye Region Segmentation in Single-Eye Videos

Will this change the current API? How?

No response

Who will benefit with this feature?

No response

Please specify the use cases for this feature

I'm conducting a study that involves analyzing eye movements and features in high-frame-rate videos. I have the following specific requirements and would greatly appreciate guidance or potential feature additions to address them: Current Dataset: 24 participants 3 videos per participant, each 10 minutes long 60 fps recordings Left eye only (no full face or other facial landmarks visible) Desired Analysis: Accurate segmentation of eye regions: pupil, iris, sclera, and whole eye contour Measurement of palpebral aperture (distance between eyelids) throughout the recordings Calculation of whole eye (contour) area over time Challenges and Questions: Is the current Facial Landmark model (which replaced the Iris Landmark model in summer 2023) capable of performing these tasks on single-eye videos? If not, is there a way to adapt or fine-tune the model for single-eye analysis without facial context? What level of accuracy can be expected for region segmentation and measurements given the high frame rate (60 fps) of our videos? Additional Considerations: The videos contain only the left eye, without other facial landmarks for reference Consistent performance is needed across a 10-minute duration for each video Any solutions should be able to handle potential variations in lighting, eye color, and eye shape across participants We would greatly appreciate any insights, suggestions, or potential feature additions that could help us achieve these analysis goals with MediaPipe. If additional information about our dataset or research requirements would be helpful, please let me know. Thank you for your consideration and assistance.

Any Other info

No response

[zxie52]

I'm conducting a study that involves analyzing eye movements and features in high-frame-rate videos. I have the following specific requirements and would greatly appreciate guidance or potential feature additions to address them:

Current Dataset:

24 participants
3 videos per participant, each 10 minutes long
60 fps recordings
Left eye only (no full face or other facial landmarks visible)
Desired Analysis:

Accurate segmentation of eye regions: pupil, iris, sclera, and whole eye contour
Measurement of palpebral aperture (distance between eyelids) throughout the recordings
Calculation of whole eye (contour) area over time
Challenges and Questions:

Is the current Facial Landmark model (which replaced the Iris Landmark model in summer 2023) capable of performing these tasks on single-eye videos?
If not, is there a way to adapt or fine-tune the model for single-eye analysis without facial context?
What level of accuracy can be expected for region segmentation and measurements given the high frame rate (60 fps) of our videos?
Additional Considerations:

The videos contain only the left eye, without other facial landmarks for reference
Consistent performance is needed across a 10-minute duration for each video
Any solutions should be able to handle potential variations in lighting, eye color, and eye shape across participants
We would greatly appreciate any insights, suggestions, or potential feature additions that could help us achieve these analysis goals with MediaPipe. If additional information about our dataset or research requirements would be helpful, please let me know.

Thank you for your consideration and assistance.

[kuaashish]

Hi @zxie52,

We do not believe the current Facial Landmark model is capable of performing well on single-eye videos. Additionally, if less than 50% of the face is visible, the model will not be able to detect it accurately. Regarding single-eye videos, we will confirm with our team and provide an update.

We do have the MediaPipe Model Maker, which offers customization options. However, it currently does not support customization for the Face Landmarker, so you will not be able to fine-tune the model at this time.

Regarding FPS-related concerns, we will get back to you with more details. However, when the Face Landmarker model is used on videos with a fully visible face and clear lighting, it performs well, especially for selfie videos. For a better understanding, please refer to the model card available here.

Thank you!!