Lloyd's Video Analysis Tool, Part 5: Major Improvements in Version 0.2.0

I have made good progress on my video analysis tool. So much so that I feel upgrading its status to "version 0.2.0" is warranted. Here are the major tasks I've tackled:

• Users can now add as many tags as they wish for each video clip;
• Users can now add a comment that is as long as they wish, extending over as many lines (i.e. paragraphs) as they wish.
• Users have the option of adding a brief description for each clip.
• I created a video clip dashboard to display all important important about the selected video clip.

I'll address each of these improvements in this blog post. First, here is a short movie of me demonstrating this prototype:

Please note that I won't be providing any LiveCode files to download as I did in previous posts for this project. For each of those posts, I had created a second and unique version of the project to make clearer the various LiveCode skills and principles I was demonstrating. The project has now progressed to the point where continuing to update and share this second LiveCode file is too time prohibitive. However, I'll continue to share important code where appropriate. And, I think those previous files will help anyone just getting into the use of video within LiveCode.

Attaching Multiple Tags for Each Video Clip

In the previous versions, the user could only add one tag per clip. I knew all along this would be insufficient. Of course, any one video clip could have many tags or no tags, so I needed to account for an uneven number in the design. I debated on the best way to approach this, deciding in the end to just have the tags continue to be added as separate items in the field "log." Doing so created a dilemma about where to put the user comments as they had been also part of this field. I decided to create a separate field for "project comments." This opened up other possibilities for comments, which I'll discuss later in this blog posting.

In previous posts I described the basics of how the tagging system works. Recall that users can create a list of unique tags. Each tag is just a copy and paste of a template button found on the "Library" card where the name and label of the button is changed to whatever tag name the user entered. I revised the script of that template button:

global gActivity
on mouseUp
   //Check to make sure that a clip has been chosen
   if the selectedtext of field "log2" is empty then
      answer "You first have to select an entry from the log."
      exit mouseUp
   end if
   put the label of me into gActivity
   put the hilitedLine of field "log2" into l // read as "line lower case L"; assume this in remaining lines
  
   //count the number of items in the line
   put the number of items of line l of field "log" into varItemsCount
  
   //check to see if tag is already there
   repeat with i = 5 to varItemsCount
      if gActivity = item i of line l of field "log" then
         set the hilitedLine of field "log2" to l
         exit mouseUp
      end if
   end repeat
   put the label of me into item varItemsCount+1 of line l of field "log"
  
   //Update the display of tags
   put "Tags: " into varShowTags
   put the number of items in line l of field "log" into varItemsCount
   if varItemsCount<5 then put "Tags: No tags for this clip" into varShowTags
   //all tags begin with item 5
   repeat with i = 5 to varItemsCount
      put varShowTags&item i of line l of field "log" into varShowTags
      if i<varItemsCount then put varShowTags&", " into varShowTags
   end repeat
   put varShowTags into line 1 of field "Tags"
  
   set the hilitedLine of field "log2" to l
  
end mouseUp

The commented lines show the main parts of this script. It begins by checking to see if the user has chosen a clip and if not, it reminds the user to do so, followed by the command "exit mouseUp." This command tells LiveCode to do just that, exit the script triggered by the mouseUp without executing any other of the lines that follow. I highlighted this line in red. (Notice that I use this command again later in the script.)

The next task is to see if the user has already chosen that tag because we obviously don't want duplicate tags listed for a clip. First,  I check to see how many items are in the highlighted line of field "log" and store this number in the local variable "varItemsCount." That number will be used for the upper limit of a repeat loop coming up, plus we will use it to determine where to put the next unique tag.

Next is the repeat loop, triggered by this line:
repeat with i = 5 to varItemsCount

The local variable i increments by one in each loop, starting with 5 and ending with the value in varItemsCount. I start with 5 because I know the first four item slots are already taken. Item 1 now contains a clip number and item 2 now contains a short description of the clip -- two things I address briefly below. Items 3 and 4 contain the starting and ending time code of the clip in user friendly terms (this was the subject of my most previous post). This repeat loop then checks to see if the tag button just pressed matches any of the tags listed in items 5 and onward. If it finds a match, it exits the mouseUp script. If it doesn't, it adds this tag as the very next item (i.e. varItemsCount + 1).

The next block of script updates the display of the tags in the video clip dashboard. This dashboard is a new feature. Let's address that next.

Video Clip Dashboard

One of the big improvements in version 0.2.0 is the video clip dashboard. Recall that I included some screen placeholders for this video dashboard in the previous versions of the prototype. Well, I finally tackled the implementation of the dashboard here.

I created four fields for the dashboard titled "time code," "Description," "tags," and "display comment." These four fields are populated with the respective information. To populate the tags field, I use another repeat loop to create a string variable called "varShowTags." This contains a list of all the tags with a comma and space in-between, all concatenated as the repeat loops works its magic.

The script ends by highlighting the video clip line in the field "log2."

Updating the Video Clip Dashboard When the User Selects a New Clip

Let's continue to look at the video clip dashboard. When the user clicks on a video entry in the field "log2," the dashboard immediately updates with the information for that clip using the following script attached to the field itself:

on mouseDown
   if the number of lines in me is 0 then exit mouseDown
   stop player "player"
   set the playSelection of player "player" to true
   put the hilitedLine of me into l
   set the currentTime of player "player" to item 3 of line l of field "log"
  
   displayClipInformation l
   displayComment l
end mouseDown

Yes, that's right, you can treat a field like a button and have events triggered if it is clicked. This short script does a lot, thanks to two new commands I created titled "displayClipInformation" and "displayComment." Commands are close relatives to functions. The idea is the same -- you are inventing your own vocabulary and adding them to LiveCode's dictionary. The difference is that a command simply does something without returning any value like a function would (review my description of functions in my previous post). The command displayClipInformation does some magic to populate the fields "time code," "Description," and "tags." The function displayComment is used to populate the field "display comment." Both functions are shown below in their entirety, but I won't try to describe them in any depth here.

Improvement to User Comments

Let's address the design behind the improvement to user comments. I knew I needed to give the user the opportunity to make in-depth comments about any of the clips. This would have to include a basic narrative structure using sentences and paragraphs, along with standard narrative symbols, such as commas and periods. It was clear that the only way to do this was by creating yet another field titled "Project Comments" to store the comments. The tricky part is connecting the right comment to the correct video clip. Since any one comment would potentially contains multiple lines, perhaps dozens, some sort of indexing system would be needed beyond just using the line number of the clip in the field "log2."

So, I used an approach that is adapted from how relational database work. I merely created an indexing system where each clip gets a unique number. I then put the term "VATComment" as item 1 on a line and the index number as item 2 on the very same line. That way, every time the user clicks on a video entry LiveCode searches the "Project Comments" field for that term as item 1 on a line. If it finds it, it then notes the index number. All lines after that and up to the next time the term VATComment is found (or the end of the field is reached), is assumed to be the entire comment for that video clip. Pretty clever if I say so myself.

I also had to design a way for a user to update a particular comment. The best strategy I came up with was to create yet another command to first "seek out and destroy"all lines associated with a particular comment when the user updates a comment, followed immediately by creating a new set of lines for the updated comment. In a sense, every update is really a new comment, as far as LiveCode is concerned. This command is titled "deleteComment" and I've also listed the entire script below. 

Final Thoughts

I also wrote a bunch of other small scripts here and there to have everything run smoothly and to catch user errors. It's really amazing the number of situations one must address when trying to anticipate a user's interaction.

One thing I have not yet done is program a way for the user to delete tags. That will come in the next version of the program. I'm currently mulling over a few ways of doing this, so you'll have to check back later to see how I decide to tackle this issue. Also, I think it is high time I address the need to let the user select any video for analysis. Yes, it is time to put my placeholder video to rest. But, you can always view it here if you miss it.

---

Appendix: Scripts for the Commands "displayClipInformation" and "displayComment"

Both scripts are defined in the card script. Both scripts take the number corresponding to the selected video clip as input.

command displayClipInformation videoEntryLine
   //Display entry information
   put videoEntryLine into l
   put item 2 of line l of field "log" into field "Description"

   //Construct time code string
   put the timescale of player "Player" into ts
   put item 3 of line l of field "log" into tcStart
   put item 4 of line l of field "log" into tcEnd
   put (tcEnd - tcStart)/ts into varCliplength
   put round(varCliplength,1) into varCliplength
   put convertTimeCode (tcStart) into tcStart2
   put convertTimeCode (tcEnd) into tcEnd2
   put "Clip length: "&tcStart2&" to "&tcEnd2&" ("&varCliplength&" sec)" into line 1 of field "Time Code"
  
   //Determine tags for display
   put "Tags: " into varShowTags
   put the number of items in line l of field "log" into varItemsCount
   if varItemsCount<5 then put "Tags: No tags for this clip" into varShowTags
   //all tags begin with item 5
   repeat with i = 5 to varItemsCount
      put varShowTags&item i of line l of field "log" into varShowTags
      if i<varItemsCount then put varShowTags&", " into varShowTags
   end repeat
   put varShowTags into line 1 of field "Tags"
end displayClipInformation

---

command displayComment videoEntryLine
   put empty into field "display comment"
   put videoEntryLine into l
   put item 1 of line videoEntryLine of field "log" into videoEntryNumber
  
   repeat with i = 1 to the number of lines in field "project comments"
      if item 1 of line i of field "project comments" is "VATComment" and item 2 of line i of field "project comments" is videoEntryNumber then
         put line i+1 of field "project comments" into line 1 of field "display comment"
         repeat with j = i+2 to the number of lines in field "project comments"
            if item 1 of line j of field "project comments" <> "VATComment" then
               put line j of field "project comments" into line (number of lines in field "display comment"+1) of field "display comment"
            else
               exit displayComment
            end if
         end repeat
         exit displayComment
      end if
   end repeat
   put "No comments yet for this video entry" into line 1 of field "display comment"
end displayComment

---

command deleteComment videoEntryLine
   //put empty into field "display comment"
   put videoEntryLine into l
   put item 1 of line videoEntryLine of field "log" into videoEntryNumber
  
   repeat with i = 1 to the number of lines in field "project comments"
      if item 1 of line i of field "project comments" is "VATComment" and item 2 of line i of field "project comments" is videoEntryNumber then
         //put line i+1 of field "project comments" into line 1 of field "display comment"
         delete line i of field "project comments"
        
         repeat with j = i to the number of lines in field "project comments"
            if item 1 of line i of field "project comments" <> "VATComment" then
               delete line i of field "project comments"
            else
               exit deleteComment
            end if
         end repeat
         exit deleteComment
      end if
   end repeat
   //put "No comments yet for this video entry" into line 1 of field "display comment"
end deleteComment

Lloyd's Video Analysis Tool, Part 4: Displaying Time Code

As we have seen so far in this series on developing a video analysis tool, LiveCode has its own unique way of tracking intervals per seconds of a video. Each video can have a different number of intervals per second and this can be determined by using the timeScale property. For the video I've been using for testing purposes in my prototype, that number has been 90,000 intervals per second.

So, using this scale, the numbers 37,800,000 to 49,950,000 would mean playing the video from the seven minute mark to the nine minute 15 second mark. The way most of us are used to seeing time code expressed is in this form: hours:minutes:seconds, or...

00:07:00 to 00:09:15

Unfortunately, LiveCode does not have a function to show the time code in this way. But, there is nothing stopping us from inventing our own function to do this. Writing custom functions is a very powerful feature of most programming languages and LiveCode is no exception. Think of it as adding to LiveCode's dictionary. Just as any human language naturally grows with use over time, so too can a computer programming language. So, we'll use this problem as an opportunity to explore how to create functions in LiveCode. I'll name this new function as "convertTimeCode." You can name functions anything you want, assuming you don't pick the name of an already existing function (some programmers add a special prefix or suffix, such as their initials, to each unique function they create to ensure no existing functions names are inadvertently used).

So, what does a function look like in use? Well, we've been using LiveCode built-in functions all along in this blog. The absolute value function is a good example. Consider the following:

   abs(-109)

This returns the number 109. Or, you could say it converts -109 to its absolute value.

Consider these two examples:

abs(598-601)
abs(601-598)

Both return a value of 3. Now consider this block of code:

put 434 into varFirst
put 822 into varSecond
abs(varFirst-varSecond)

This will return a value of 388.

Similarly, let's imagine I've already created the new function "convertTimeCode" and entered the following as part of some scripts:

   convertTimeCode(49950000)
   convertTimeCode(324990000)

These each return values, respectively, of "00:09:15" and "03:00:11".

(Of course, these two examples assume a timeScale value of 90,000; remember, the timeScale value can be different depending on the video. So, we just need to always check for the timeScale when writing our scripts.)

Creating Our Own LiveCode Function

The complete script of the function I created is at the bottom of this post. I'll explain briefly how it works in a moment, but here is its basic construction:

function convertTimeCode ct
     //all of the script that makes the function work goes here
     return myTimeCode
end convertTimeCode

The number that is entered in parentheses when the function is called is put into "ct," a local variable that I get to name. (This was meant to be short for "time code," but I guess I got the letters backwards when I wrote the function!) The script that makes the function work eventually put the final value of the time code into a variable called "myTimeCode." Its value must look like the time code described above (e.g. 01:34:14).

I don't think a full explanation of how this function actually works is needed here. In short, the function takes the number of intervals and converts it to seconds. This is then divided by 3600 -- the number of seconds in an hour -- to figure out how many hours there are. The remainder is then divided by 60 -- the number of seconds in a minute -- to figure out how many remaining minutes there are. The remainder is the number of seconds. I relied on LiveCode's trunc function (short for truncate) for these steps. When you have numbers like 29.35 or 31.78, the trunc function strips away everything to the right of the decimal point, leaving just the integer, such as 29 and 31. No rounding occurs.

Along the way, the script adds extra zeros if and when they are necessary. For example, if there are only 9 minutes for the middle part of the time code, an extra zero is added in front of the 9 (e.g. "09"). Finally, colons are added where needed.

Figuring all this out was just a fun little logic puzzle for me. I'm sure there are probably more elegant ways to do this, but hey, it works.

Displaying the Time Code to the User

Great, we have a function that converts LiveCode's intervals into time code that a person can interpret. Now, what should we do with it? We need to display this time code and have the user interact with it. Of course, this nicely formatted time code isn't what LiveCode uses to actually play video snippets. So, we will continue to need the original video interval information. As in many of my previous LiveCode projects, the strategy here once again points to the use of what I call "Shadow Fields." These are fields that are actually hidden from the user's view, but do work behind the scenes.

I've copied and pasted the field "log" and named this new field simply "log2." I'll put all of the time code data inside log2 in parallel with the interval data in the original log. For now, I'm displaying both so you can see the parallel output, though I'll eventually hide the field "log" (the field "log2" is on the left and the field "log" is on the right):

 LiveCode file

[ Get the free LiveCode Community version. ]

As you can see, the two lines of the two fields match, or correspond, to each other. All I need to do now is focus the user's interaction on log2 instead of log. But, I have to be careful to use the original log when sending instructions to the movie player. This is easily done because I can refer to the same line number in each field. I just have to be careful when I reference log2 versus log. As an example, here is the revised script for the "Update Start" button:

on mouseUp
   put the hilitedLine of field "log2" into l //note: this is a lower-case L
   put the currentTime of player "player" into ct
   //error trap - starting point must be before ending point
   if ct > item 2 of line l of field "log" then //note: this is a lower-case L
      answer error "Not Allowed! Starting point must come before the ending point!" with "OK"
      exit mouseUp
   end if
   put ct into item 1 of line l of field "log" //note: read this as "into item one of line [lower-case L]"
   put convertTimeCode(ct) into item 1 of line l of field "log2" //ditto
   set the hilitedLine of field "log2" to l //note: this is a lower-case L
end mouseUp

First, notice my use of the convertTimeCode function in the second-to-last line of the script (not counting the "on mouseUp" and "end mouseUp" lines). (Note: Sorry the differences between the number one and the lower-case L are not more readily apparent. I hope my notes in red make this clearer.)

Second, notice how I switch back and forth between referencing log2 versus log. I do the same in the other buttons in the blue editing zone.

One more quick enhancement: Adding an Alert Box

Every time a new entry is added to the log, the previously selected entry is un-highlighted. This causes a problem if the user then immediately clicks on the play button in the editing zone. So, we need to communicate a reminder to the user that they first need to highlight one of the video entries first. We could dim out the play button if nothing is highlighted (which, frankly, sounds like the way to go), but for now I've just added a little alert box in the button "Play entry." Here's the script I added to that button:

   if the selectedtext of field "log2" is empty then
      answer "You first have to select an entry from the log."
      exit mouseUp
   end if

Things are coming along very nicely for this video analysis tool project.

---

Appendix

Here's the script for my "convertTimeCode" function - impressive, isn't it?

function convertTimeCode ct
   put the timescale of player "Player" into ts
   //Convert intervals into seconds
   put round((ct)/ts) into s
   //Determine number of hours and subtract that number of seconds from total
   if s<3600 then put "00" into myTimeCode
   if s>=3600 then
      put trunc(s/3600) into h
      put s-(h*3600) into s
      if h<10 then
         put "0"&h into myTimeCode
      else
         put h into myTimeCode
      end if
   end if
   //Determine number of minutes
   if s<60 then put myTimeCode&":00" into myTimeCode
   if s>=60 then
      put trunc(s/60) into m
      put s-(m*60) into s
      if m<10 then
         put myTimeCode&":0"&m into myTimeCode
      else
         put myTimeCode&":"&m into myTimeCode
      end if
   end if
   //Determine number of seconds
   if s>=10 then put myTimeCode&":"&s into myTimeCode
   if s<10 then put myTimeCode&":0"&s into myTimeCode
   return myTimeCode
end convertTimeCode