Overview  Package   Class  Tree  Deprecated  Index  Help 
 PREV CLASS   NEXT CLASS FRAMES    NO FRAMES    
SUMMARY: NESTED | FIELD | CONSTR | METHOD DETAIL: FIELD | CONSTR | METHOD

com.jogamp.opengl.util.av
Interface GLMediaPlayer

All Superinterfaces:
TextureSequence
public interface GLMediaPlayer
extends TextureSequence

GLMediaPlayer interface specifies a TextureSequence state machine using a multiplexed audio/video stream as it's source.

Audio maybe supported and played back internally or via an AudioSink implementation.

Audio and video streams can be selected or muted via initStream(URI, int, int, int) using the appropriate stream id's.

Camera input can be selected using the CameraInputScheme URI.

StreamWorker Decoding Thread

Most of the stream processing is performed on the decoding thread, a.k.a. StreamWorker:

StreamWorker generates it's own GLContext, shared with the one passed to initGL(GL). The shared GLContext allows the decoding thread to push the video frame data directly into the designated TextureFrame, later returned via getNextTexture(GL) and used by the user.

StreamWorker Error Handling

Caught exceptions on StreamWorker are delivered as GLMediaPlayer.StreamExceptions, which either degrades the GLMediaPlayer.State to GLMediaPlayer.State.Uninitialized or GLMediaPlayer.State.Paused.

An occurring GLMediaPlayer.StreamException triggers a EVENT_CHANGE_ERR event, which can be listened to via GLMediaPlayer.GLMediaEventListener.attributesChanged(GLMediaPlayer, int, long).

An occurred GLMediaPlayer.StreamException can be read via getStreamException().

GLMediaPlayer Lifecycle

Action GLMediaPlayer.State Before GLMediaPlayer.State After Event
initStream(URI, int, int, int) Uninitialized Initialized1, Uninitialized EVENT_CHANGE_INIT or ( EVENT_CHANGE_ERR + EVENT_CHANGE_UNINIT )
initGL(GL) Initialized Paused, , Uninitialized EVENT_CHANGE_PAUSE or ( EVENT_CHANGE_ERR + EVENT_CHANGE_UNINIT )
play() Paused Playing EVENT_CHANGE_PLAY
pause(boolean) Playing Paused EVENT_CHANGE_PAUSE
seek(int) Paused, Playing Paused, Playing none
getNextTexture(GL) Paused, Playing Paused, Playing none
getLastTexture() Paused, Playing Paused, Playing none
END_OF_STREAM Playing Paused EVENT_CHANGE_EOS + EVENT_CHANGE_PAUSE
GLMediaPlayer.StreamException ANY Paused, Uninitialized EVENT_CHANGE_ERR + ( EVENT_CHANGE_PAUSE or EVENT_CHANGE_UNINIT )
destroy(GL) ANY Uninitialized EVENT_CHANGE_UNINIT

Audio and video Stream IDs

value request get
STREAM_ID_NONE mute not available
STREAM_ID_AUTO auto unspecified
≥0 specific stream specific stream

Current implementations (check each API doc link for details):

Implementations of this interface must implement: 

    public static final boolean isAvailable();
to be properly considered by GLMediaPlayerFactory.create(ClassLoader, String) and GLMediaPlayerFactory.createDefault().

Timestamp Accuracy

Timestamp type and value range has been chosen to suit embedded CPUs and characteristics of audio and video streaming. See TimeFrameI.

Audio and video synchronization

The class follows a passive A/V synchronization pattern. Audio is being untouched, while getNextTexture(GL) delivers a new video frame only, if its timestamp is less than MAXIMUM_VIDEO_ASYNC ahead of time. If its timestamp is more than MAXIMUM_VIDEO_ASYNC ahead of time, the previous frame is returned. If its timestamp is more than MAXIMUM_VIDEO_ASYNC after time, the frame is dropped and the next frame is being fetched.

https://en.wikipedia.org/wiki/Audio_to_video_synchronization 

   d_av = v_pts - a_pts;

Recommendation of audio/video pts time lead/lag at production:

Recommendation of av pts time lead/lag at presentation:

Test Streams

Big Buck Bunny 24f 16:9
Big Buck Bunny320ph264aac 48000Hz 2 chanhttp://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4
Big Buck Bunny720pmpeg4ac3 48000Hz 5.1 chanhttp://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_surround.avi
Big Buck Bunny720pmsmpeg4v2mp3 48000Hz 2 chanhttp://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_stereo.avi
Big Buck Bunny720ptheoravorbis 48000Hz 2 chanhttp://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_stereo.ogg
Big Buck Bunny1080pmpeg4ac3 48000Hz 5.1 chanhttp://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_surround.avi
WebM/Matroska (vp8/vorbis)
Big Buck Bunny Trailer640pvp8vorbis 44100Hz 1 chanhttp://video.webmfiles.org/big-buck-bunny_trailer.webm
Elephants Dream540pvp8vorbis 44100Hz 1 chanhttp://video.webmfiles.org/elephants-dream.webm
You Tube http/rtsp
Sintelhttp://www.youtube.com/watch?v=eRsGyueVLvQrtsp://v3.cache1.c.youtube.com/CiILENy73wIaGQn0LpXnygYbeRMYDSANFEgGUgZ2aWRlb3MM/0/0/0/video.3gp
Audio/Video Sync
Five-minute-sync-test1080phttps://www.youtube.com/watch?v=szoOsG9137Urtsp://v7.cache8.c.youtube.com/CiILENy73wIaGQm133VvsA46sxMYDSANFEgGUgZ2aWRlb3MM/0/0/0/video.3gp
Audio-Video-Sync-Test-Calibration-23.98fps-24fpshttps://www.youtube.com/watch?v=cGgf_dbDMsw
sound_in_sync_testhttps://www.youtube.com/watch?v=O-zIZkhXNLE
Nested Class Summary
static interface GLMediaPlayer.GLMediaEventListener
          
static class GLMediaPlayer.State
          See Lifecycle.
static class GLMediaPlayer.StreamException
          A StreamException encapsulates a caught exception in the decoder thread, a.k.a StreamWorker, see See StreamWorker Error Handling.
 
Nested classes/interfaces inherited from interface com.jogamp.opengl.util.texture.TextureSequence
TextureSequence.TexSeqEventListener<T extends TextureSequence>, TextureSequence.TextureFrame
 
Field Summary
static String CameraInputScheme
          URI scheme name "camera" for camera input.
static String CameraPropHeight
          Camera property "height".
static String CameraPropRate
          Camera property "rate".
static String CameraPropSizeS
          Camera property "size", size as string, e.g.
static String CameraPropWidth
          Camera property "width".
static boolean DEBUG
           
static boolean DEBUG_NATIVE
           
static int MAXIMUM_VIDEO_ASYNC
          Maximum video frame async of 22 milliseconds.
static int STREAM_ID_AUTO
          Constant -1 for auto or unspecified.
static int STREAM_ID_NONE
          Constant -2 for mute or not available.
static int TEXTURE_COUNT_DEFAULT
          Default texture count, value 4.
static int TEXTURE_COUNT_MIN
          Minimum texture count, value 1.
 
Fields inherited from interface com.jogamp.opengl.util.texture.TextureSequence
sampler2D, samplerExternalOES
 
Method Summary
 void addEventListener(GLMediaPlayer.GLMediaEventListener l)
          Adds a GLMediaPlayer.GLMediaEventListener to this player.
 Object attachObject(String name, Object obj)
          Attaches the user object for the given name.
 GLMediaPlayer.State destroy(GL gl)
          Releases the GL, stream and other resources, including attached user objects.
 Object detachObject(String name)
          Detaches the user object for the given name.
 int getAID()
          Return the audio stream id, see audio and video Stream IDs.
 Object getAttachedObject(String name)
          Returns the attached user object for the given name.
 int getAudioBitrate()
          Warning: Optional information, may not be supported by implementation.
 String getAudioCodec()
          Warning: Optional information, may not be supported by implementation.
 int getAudioFrames()
          Warning: Optional information, may not be supported by implementation.
 int getAudioPTS()
           
 AudioSink getAudioSink()
          If implementation uses a AudioSink, it's instance will be returned.
 float getAudioVolume()
          Returns the audio volume.
 int getDecodedFrameCount()
           
 int getDuration()
           
 GLMediaPlayer.GLMediaEventListener[] getEventListeners()
          Return all GLMediaPlayer.GLMediaEventListener of this player.
 float getFramerate()
          Warning: Optional information, may not be supported by implementation.
 int getHeight()
          Returns the height of the video.
 TextureSequence.TextureFrame getLastTexture()
          Returns the last updated texture.
 TextureSequence.TextureFrame getNextTexture(GL gl)
          Returns the next texture to be rendered.
 String getPerfString()
          Returns a string represantation of this player's performance values.
 float getPlaySpeed()
          Returns the playback speed.
 int getPresentedFrameCount()
           
 GLMediaPlayer.State getState()
          See Lifecycle.
 long getStreamBitrate()
          Warning: Optional information, may not be supported by implementation.
 GLMediaPlayer.StreamException getStreamException()
          Returns the GLMediaPlayer.StreamException caught in the decoder thread, or null if none occured.
 int getTextureCount()
           
 URI getURI()
          Return the stream location, as set by initStream(URI, int, int, int).
 int getVID()
          Return the video stream id, see audio and video Stream IDs.
 int getVideoBitrate()
          Warning: Optional information, may not be supported by implementation.
 String getVideoCodec()
          Warning: Optional information, may not be supported by implementation.
 int getVideoFrames()
          Warning: Optional information, may not be supported by implementation.
 int getVideoPTS()
           
 int getWidth()
          Returns the width of the video.
 void initGL(GL gl)
          Initializes OpenGL related resources.
 void initStream(URI streamLoc, int vid, int aid, int textureCount)
          Issues asynchronous stream initialization.
 boolean isGLOriented()
          Returns true if the video frame is oriented in OpenGL's coordinate system, origin at bottom left.
 GLMediaPlayer.State pause(boolean flush)
          Pauses the StreamWorker decoding thread.
 GLMediaPlayer.State play()
          Starts or resumes the StreamWorker decoding thread.
 void removeEventListener(GLMediaPlayer.GLMediaEventListener l)
          Removes a GLMediaPlayer.GLMediaEventListener to this player.
 int seek(int msec)
          Seeks to the new absolute position.
 boolean setAudioVolume(float v)
          Sets the audio volume, [0f..1f].
 boolean setPlaySpeed(float rate)
          Sets the playback speed.
 void setTextureMinMagFilter(int[] minMagFilter)
          Sets the texture min-mag filter, defaults to GL.GL_NEAREST.
 void setTextureUnit(int u)
          Sets the texture unit.
 void setTextureWrapST(int[] wrapST)
          Sets the texture min-mag filter, defaults to GL.GL_CLAMP_TO_EDGE.
 String toString()
          Returns a string represantation of this player, incl.
 
Methods inherited from interface com.jogamp.opengl.util.texture.TextureSequence
getRequiredExtensionsShaderStub, getTextureLookupFragmentShaderImpl, getTextureLookupFunctionName, getTextureMinMagFilter, getTextureSampler2DType, getTextureTarget, getTextureUnit, getTextureWrapST
 

Field Detail

DEBUG

static final boolean DEBUG

DEBUG_NATIVE

static final boolean DEBUG_NATIVE

TEXTURE_COUNT_DEFAULT

static final int TEXTURE_COUNT_DEFAULT
Default texture count, value 4.

See Also:
Constant Field Values

TEXTURE_COUNT_MIN

static final int TEXTURE_COUNT_MIN
Minimum texture count, value 1. Using the minimum texture count disables multi-threaded decoding.

See Also:
Constant Field Values

STREAM_ID_NONE

static final int STREAM_ID_NONE
Constant -2 for mute or not available. See Audio and video Stream IDs.

See Also:
Constant Field Values

STREAM_ID_AUTO

static final int STREAM_ID_AUTO
Constant -1 for auto or unspecified. See Audio and video Stream IDs.

See Also:
Constant Field Values

CameraInputScheme

static final String CameraInputScheme
URI scheme name "camera" for camera input. E.g. camera:/0 for the 1st camera device.

The URI path is being used to identify the camera (ID), where the root fwd-slash is being cut-off.

The ID is usually an integer value indexing the camera ranging from [0..max-number].

The URI query is used to pass options to the camera using ; as the separator. The latter avoids trouble w/ escaping. 

    camera:/
    camera://somewhere/
    camera://somewhere/?width=640;height=480;rate=15
    camera://somewhere/?size=640x480;rate=15
 
  URI: [scheme:][//authority][path][?query][#fragment]
  w/ authority: [user-info@]host[:port]
  Note: 'path' starts w/ fwd slash

See Also:
Constant Field Values

CameraPropSizeS

static final String CameraPropSizeS
Camera property "size", size as string, e.g. 1280x720, hd720. May not be supported on all platforms. See CameraInputScheme.

See Also:
Constant Field Values

CameraPropWidth

static final String CameraPropWidth
Camera property "width". See CameraInputScheme.

See Also:
Constant Field Values

CameraPropHeight

static final String CameraPropHeight
Camera property "height". See CameraInputScheme.

See Also:
Constant Field Values

CameraPropRate

static final String CameraPropRate
Camera property "rate". See CameraInputScheme.

See Also:
Constant Field Values

MAXIMUM_VIDEO_ASYNC

static final int MAXIMUM_VIDEO_ASYNC
Maximum video frame async of 22 milliseconds.

See Also:
Constant Field Values
Method Detail

getTextureCount

int getTextureCount()

setTextureUnit

void setTextureUnit(int u)
Sets the texture unit. Defaults to 0.

setTextureMinMagFilter

void setTextureMinMagFilter(int[] minMagFilter)
Sets the texture min-mag filter, defaults to GL.GL_NEAREST.

setTextureWrapST

void setTextureWrapST(int[] wrapST)
Sets the texture min-mag filter, defaults to GL.GL_CLAMP_TO_EDGE.

initStream

void initStream(URI streamLoc,
                int vid,
                int aid,
                int textureCount)
                throws IllegalStateException,
                       IllegalArgumentException
Issues asynchronous stream initialization.

Lifecycle: GLMediaPlayer.State.Uninitialized -> GLMediaPlayer.State.Initialized1 or GLMediaPlayer.State.Uninitialized

GLMediaPlayer.State.Initialized is reached asynchronous, i.e. user gets notified via attributesChanges(..).

A possible caught asynchronous GLMediaPlayer.StreamException while initializing the stream off-thread will be thrown at initGL(GL).

Muted audio can be achieved by passing STREAM_ID_NONE to aid.

Muted video can be achieved by passing STREAM_ID_NONE to vid, in which case textureCount is ignored as well as the passed GL object of the subsequent initGL(GL) call.

Parameters:
streamLoc - the stream location
vid - video stream id, see audio and video Stream IDs
aid - video stream id, see audio and video Stream IDs
textureCount - desired number of buffered textures to be decoded off-thread, will be validated by implementation. The minimum value is TEXTURE_COUNT_DEFAULT. Ignored if video is muted.
Throws:
IllegalStateException - if not invoked in GLMediaPlayer.State.Uninitialized
IllegalArgumentException - if arguments are invalid

getStreamException

GLMediaPlayer.StreamException getStreamException()
Returns the GLMediaPlayer.StreamException caught in the decoder thread, or null if none occured.

Method clears the cached GLMediaPlayer.StreamException, hence an immediate subsequent call will return null.

See Also:
GLMediaPlayer.GLMediaEventListener.EVENT_CHANGE_ERR, GLMediaPlayer.StreamException

initGL

void initGL(GL gl)
            throws IllegalStateException,
                   GLMediaPlayer.StreamException,
                   GLException
Initializes OpenGL related resources.

Lifecycle: GLMediaPlayer.State.Initialized -> GLMediaPlayer.State.Paused or GLMediaPlayer.State.Initialized

Argument gl is ignored if video is muted, see initStream(URI, int, int, int).

Parameters:
gl - current GL object. Maybe null, for audio only.
Throws:
IllegalStateException - if not invoked in GLMediaPlayer.State.Initialized.
GLMediaPlayer.StreamException - forwarded from the off-thread stream initialization
GLException - in case of difficulties to initialize the GL resources

getAudioSink

AudioSink getAudioSink()
If implementation uses a AudioSink, it's instance will be returned.

The AudioSink instance is available after initStream(URI, int, int, int), if used by implementation.

destroy

GLMediaPlayer.State destroy(GL gl)
Releases the GL, stream and other resources, including attached user objects.

Lifecycle: ANY -> GLMediaPlayer.State.Uninitialized

setPlaySpeed

boolean setPlaySpeed(float rate)
Sets the playback speed.

To simplify test, play speed is normalized, i.e.

Returns:
true if successful, otherwise false, i.e. due to unsupported value range of implementation.

getPlaySpeed

float getPlaySpeed()
Returns the playback speed.

setAudioVolume

boolean setAudioVolume(float v)
Sets the audio volume, [0f..1f].

To simplify test, volume is normalized, i.e.

Returns:
true if successful, otherwise false, i.e. due to unsupported value range of implementation.

getAudioVolume

float getAudioVolume()
Returns the audio volume.

play

GLMediaPlayer.State play()
Starts or resumes the StreamWorker decoding thread.

Lifecycle: GLMediaPlayer.State.Paused -> GLMediaPlayer.State.Playing

pause

GLMediaPlayer.State pause(boolean flush)
Pauses the StreamWorker decoding thread.

Lifecycle: GLMediaPlayer.State.Playing -> GLMediaPlayer.State.Paused

If a new frame is desired after the next play() call, e.g. to make a snapshot of a camera input stream, flush shall be set to true.

Parameters:
flush - if true flushes the video and audio buffers, otherwise keep them intact.

seek

int seek(int msec)
Seeks to the new absolute position. The StreamWorker decoding thread is paused while doing so and the A/V buffers are flushed.

Allowed in state GLMediaPlayer.State.Playing and GLMediaPlayer.State.Paused, otherwise ignored, see Lifecycle.

Parameters:
msec - absolute desired time position in milliseconds
Returns:
time current position in milliseconds, after seeking to the desired position

getState

GLMediaPlayer.State getState()
See Lifecycle.

Returns:
the current state, either GLMediaPlayer.State.Uninitialized, GLMediaPlayer.State.Initialized, GLMediaPlayer.State.Playing or GLMediaPlayer.State.Paused

getVID

int getVID()
Return the video stream id, see audio and video Stream IDs.

getAID

int getAID()
Return the audio stream id, see audio and video Stream IDs.

getDecodedFrameCount

int getDecodedFrameCount()
Returns:
the current decoded frame count since play() and seek(int) as increased by getNextTexture(GL) or the decoding thread.

getPresentedFrameCount

int getPresentedFrameCount()
Returns:
the current presented frame count since play() and seek(int) as increased by getNextTexture(GL) for new frames.

getVideoPTS

int getVideoPTS()
Returns:
current video presentation timestamp (PTS) in milliseconds of getLastTexture()

getAudioPTS

int getAudioPTS()
Returns:
current audio presentation timestamp (PTS) in milliseconds.

getLastTexture

TextureSequence.TextureFrame getLastTexture()
                                            throws IllegalStateException
Returns the last updated texture.

In case the instance is just initialized, it shall return a TextureFrame object with valid attributes. The texture content may be undefined until the first call of TextureSequence.getNextTexture(GL).

Not blocking.

See audio and video synchronization.

Specified by:
getLastTexture in interface TextureSequence
Throws:
IllegalStateException - if not invoked in GLMediaPlayer.State.Paused or GLMediaPlayer.State.Playing

getNextTexture

TextureSequence.TextureFrame getNextTexture(GL gl)
                                            throws IllegalStateException
Returns the next texture to be rendered.

Implementation shall return the next frame if available, may block if a next frame may arrive soon. Otherwise implementation shall return the last frame.

Shall return null in case no next or last frame is available.

In case the current state is not GLMediaPlayer.State.Playing, getLastTexture() is returned.

See audio and video synchronization.

Specified by:
getNextTexture in interface TextureSequence
Throws:
IllegalStateException - if not invoked in GLMediaPlayer.State.Paused or GLMediaPlayer.State.Playing
See Also:
addEventListener(GLMediaEventListener), GLMediaEventListener#newFrameAvailable(GLMediaPlayer, TextureFrame, long)

getURI

URI getURI()
Return the stream location, as set by initStream(URI, int, int, int).

getVideoCodec

String getVideoCodec()
Warning: Optional information, may not be supported by implementation.

Returns:
the code of the video stream, if available

getAudioCodec

String getAudioCodec()
Warning: Optional information, may not be supported by implementation.

Returns:
the code of the audio stream, if available

getVideoFrames

int getVideoFrames()
Warning: Optional information, may not be supported by implementation.

Returns:
the total number of video frames

getAudioFrames

int getAudioFrames()
Warning: Optional information, may not be supported by implementation.

Returns:
the total number of audio frames

getDuration

int getDuration()
Returns:
total duration of stream in msec.

getStreamBitrate

long getStreamBitrate()
Warning: Optional information, may not be supported by implementation.

Returns:
the overall bitrate of the stream.

getVideoBitrate

int getVideoBitrate()
Warning: Optional information, may not be supported by implementation.

Returns:
video bitrate

getAudioBitrate

int getAudioBitrate()
Warning: Optional information, may not be supported by implementation.

Returns:
the audio bitrate

getFramerate

float getFramerate()
Warning: Optional information, may not be supported by implementation.

Returns:
the framerate of the video

isGLOriented

boolean isGLOriented()
Returns true if the video frame is oriented in OpenGL's coordinate system, origin at bottom left.

Otherwise returns false, i.e. video frame is oriented origin at top left.

false is the default assumption for videos, but user shall not rely on.

false GL orientation leads to Texture.getMustFlipVertically() == true, as reflected by all TextureFrame's Textures retrieved via getLastTexture() or getNextTexture(GL).

getWidth

int getWidth()
Returns the width of the video.

getHeight

int getHeight()
Returns the height of the video.

toString

String toString()
Returns a string represantation of this player, incl. state and audio/video details.

Overrides:
toString in class Object

getPerfString

String getPerfString()
Returns a string represantation of this player's performance values.

addEventListener

void addEventListener(GLMediaPlayer.GLMediaEventListener l)
Adds a GLMediaPlayer.GLMediaEventListener to this player.

removeEventListener

void removeEventListener(GLMediaPlayer.GLMediaEventListener l)
Removes a GLMediaPlayer.GLMediaEventListener to this player.

getEventListeners

GLMediaPlayer.GLMediaEventListener[] getEventListeners()
Return all GLMediaPlayer.GLMediaEventListener of this player.

getAttachedObject

Object getAttachedObject(String name)
Returns the attached user object for the given name.

attachObject

Object attachObject(String name,
                    Object obj)
Attaches the user object for the given name. Returns the previously set object, may be null.

detachObject

Object detachObject(String name)
Detaches the user object for the given name. Returns the previously set object, may be null.

Overview  Package   Class  Tree  Deprecated  Index  Help 
 PREV CLASS   NEXT CLASS FRAMES    NO FRAMES    
SUMMARY: NESTED | FIELD | CONSTR | METHOD DETAIL: FIELD | CONSTR | METHOD
Copyright 2010 JogAmp Community.