Share Your Code

SoundTextSync - Read-it-to-me Story Books and interactive text

Posted by DavidBFox, Posted on November 23, 2010

Inspired by interactive story books for kids, like Broderbund's old Living Book titles from the 1990s. When the narrator reads the text, the words light up or are highlighted as they are spoken.

After the narrator finishes reading, you can tap on any word and hear that word spoken.

I started with some sample audio and code contributed by cmontesino (see https://developer.anscamobile.com/forum/2010/11/14/read-it-me-story-books-and-interactive-text for our discussion). I imported the audio into Audacity (an open source sound editing program) added labels to each section of sound corresponding to a word. Label information timings were then exported from Audacity and used as the bases for the table in this demo.

I've included the Audacity files (the .aup file that's included, along with the folder) so you can try this out and see how it works. Also included are the resulting caf files (as well as mp3 files if you want to try this on Android devices).

For this demo, you can click on the black box to the left of the text to play the entire recording (with word highlighting synced to the audio), or you can click on each word to hear it by itself and see it highlighted.

For a real production, I would record each page's contents twice... once just read straight through (as we have here), and once with clear pauses between each word so they're enunciated clearly and don't blend into the next word.

I used cmontesino's idea of switching between different colored text objects to highlight each word, but you could also highlight behind the text using the width of the word. And if we ever get fancier ways to display text (like with outlines or glows), we could come up with fancier ways to display.

Note that on the Corona simulator, the sounds for the individual touched words is much quieter than when the full sentence is read. I think this must be a simulator bug. Sounds fine on the iPad.

To download:
http://www.ElectricEggplant.com/corona/SoundTextSync.zip

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
-- 
-- Abstract: Sound synchronized with text highlighting.
-- Intended use is for "Read it to me" story book apps.
-- 
-- Version: 1.1 - November 22, 2010
-- 
-- David Fox, Electric Eggplant - http://ElectricEggplant.com
 
--  There are two functions here... clicking on the black square to the
--  left of the text triggers the voice reading the entire text. Each
--  word is highlighted as it is spoken. Also you can tap on any word
--  to hear it spoken by itself. Since I worked from the original audio
--  file, this is not ideal -- words are sometimes cut off or not
--  clear. Best practice would be to have two recordings. One with the
--  sentences being read normally. And another with a space between
--  each word so they can easily be cut up. For the latter, the code
--  would have to be modified to include timings for the second set of
--  recordings as well.
--
--  Included are both MP3 and CAF versions. Both will play in the simulator.
--  Use CAF for iOS devices, MP3 for Android devices. I've included the
--  Audacity file so you can see how the labels were added.
--
--  Thank you to cmontesino for providing the recording and original code,
--  and for jayantv for inspiration!
 
 
-- load sound so there's no delay on button press (is there a better way to do this?)
media.playSound("1.caf")
media.stopSound()
 
local bg = display.newRect( 0, 0, display.contentWidth, display.contentHeight )
bg:setFillColor(255,255,255)
blackText={}
redText = {}
local soundLength = 0
 
-- Data is obtained by loading the full audio track into Audacity (freeware
-- sound editing program), selecting each word and creating a label. The word spoken
-- is the name for the label. Then Export Labels to create a text file with the data.
-- Times are in seconds, so multiply by 1000 to convert to milliseconds.
-- Add newline=true where you want a CRLF (so the following line starts on a new line)
local voice = {
  {start=0.359055, out=0.643383, name="Twas"},
  {start=0.643383, out=0.752740, name="the"},
  {start=0.752740, out=1.210217, name="night"},
  {start=1.210217, out=1.600257, name="before"},
  {start=1.600257, out=2.456886, name="Christmas"},
  {start=2.456886, out=2.646438, name="when"},
  {start=2.646438, out=3.018252, name="all"},
  {start=3.018252, out=3.258837, name="through"},
  {start=3.258837, out=3.384598, name="the"},
  {start=3.384598, out=3.924092, name="house,", newline=true},
  {start=4.441715, out=4.576588, name="Not"},
  {start=4.576588, out=4.702349, name="a"},
  {start=4.702349, out=5.088743, name="creature"},
  {start=5.088743, out=5.316570, name="was"},
  {start=5.316570, out=5.892517, name="stirring,"},
  {start=6.016455, out=6.236991, name="not"},
  {start=6.236991, out=6.464818, name="even"},
  {start=6.464818, out=6.555038, name="a"},
  {start=6.555038, out=7.217559, name="mouse.", newline=true},
  {start=7.879168, out=7.964831, name="The"},
  {start=7.964831, out=8.606392, name="stockings"},
  {start=8.606392, out=8.741266, name="were"},
  {start=8.741266, out=9.173226, name="hung"},
  {start=9.173226, out=9.360955, name="by"},
  {start=9.360955, out=9.459376, name="the"},
  {start=9.459376, out=9.935079, name="chimney"},
  {start=9.935079, out=10.164729, name="with"},
  {start=10.164729, out=10.634963, name="care,", newline=true},
  {start=11.269234, out=11.469721, name="In"},
  {start=11.469721, out=12.042023, name="hopes"},
  {start=12.042023, out=12.211526, name="that"},
  {start=12.211526, out=13.124657, name="St. Nicholas"},
  {start=13.124657, out=13.543858, name="soon"},
  {start=13.543858, out=13.749814, name="would"},
  {start=13.749814, out=13.875574, name="be"},
  {start=13.875574, out=14.287486, name="there."},
}
 
-- Using the voice table, display the text by creating an object for each word
local function displayText(params)
  local x,y,color,alpha = params.x, params.y, params.color, params.alpha
  local xOffset = 0
  local words={}
  local fontSize = 32
  local lineHeight = fontSize*1.33
  local space = fontSize/5
 
  for i = 1,#voice do
    words[i] = display.newText(voice[i].name, x+xOffset, y, "Baskerville", fontSize)
    words[i]:setTextColor( color[1],color[2],color[3])
    words[i].alpha = alpha
    -- convert to lower case and remove punctuation from name so we can use it 
    -- to grab the correct audio file later
    words[i].name = string.lower(string.gsub(voice[i].name, "['., ]", ""))
    words[i].id = i
    --  calculate the duration of each word
    words[i].dur = (voice[i].out - voice[i].start) * 1000
    if params.addListner then
      words[i]:addEventListener( "tap", speakWord )
    end
    xOffset = xOffset + words[i].width + space
    if voice[i].newline then y = y + lineHeight; xOffset = 0 end
  end
  soundLength = voice[#voice].out*1000
  return words
end
 
-- Add a button to start the talking
local function displayButton(params)
  local x,y,w,h,color = params.x, params.y, params.w, params.h, params.color
  local rect = display.newRect(x, y, w, h)
  rect:setFillColor(color[1],color[2],color[3])
  return rect
end
 
-- not currently being called
local function stopNBC()
  media.stopSound("1.caf")
end
 
-- The button was pressed, so start talking. Highlight each word as it's spoken.
-- trans1 is the delay in milliseconds for the fade to red as the word is spoken
-- trans2 is the delay in milliseconds for the fade back to black after the word is spoken
-- use a shorter trans1 value for snappier response. Longer trans2 to make the fade
-- back to black more fluid.
local function saySentence(event)
  local delay1, delay2, trans1, trans2 = 0,0,200,400
  media.playSound("1.caf")
 
  -- switch to red button so it's not touchable
  transition.dissolve(blackButton,redButton,trans1,0)
  transition.dissolve(redButton,blackButton,trans1,soundLength+trans2)
 
  for i = 1,#voice do
    -- start transition early so it's full red by the time the word is spoken
    delay1 = voice[i].start*1000 - trans1
    if delay1 <0 then delay1 = 0 end
    -- add extra time at the end so we never finish before the fade is complete
    delay2 = voice[i].out*1000 + trans2
    transition.dissolve(blackText[i],redText[i],trans1,delay1)
    transition.dissolve(redText[i],blackText[i],trans2,delay2)
  end
end
 
-- A word was touched, so say it now. Disabled during speech.
function speakWord( event )
  local word = event.target
   local id, name, dur = word.id, word.name, word.dur
   local trans = 200
   -- was the sentence button pushed or this word already active? If so, return now.
   if blackButton.alpha == 0 or word.alpha ~= 1 then return end
   -- make sure the duration is at least longer than 2 transition times
   dur = dur + 2*trans
  media.playEventSound("snippets/"..name..".caf")
  transition.dissolve(word,redText[id],trans,0)
  transition.dissolve(redText[id],word,trans,dur)
  return true
end
 
-- initialize
redButton = displayButton{x=40,y=210,w=40,h=40,color={255,0,0}}
blackButton = displayButton{x=40,y=210,w=40,h=40,color={60,60,60}}
blackButton:addEventListener( "tap", saySentence )
 
blackText = displayText{x=100,y=200,color={60,60,60},alpha=1,addListner=true}
redText = displayText{x=100,y=200,color={255,0,0},alpha=0}

Compatibility: 
Corona 2.0

Replies

alcamie
User offline. Last seen 1 year 30 weeks ago. Offline
Joined: 25 Sep 2010

I'm trying to use this for a kids book using the Director class and I'm having trouble adding and removing the soundTextSync and recreating per screen.

Do you have any hints and tips on how to use this with the Director Class?

Best regards,
Chris

DavidBFox
User offline. Last seen 2 weeks 14 hours ago. Offline
Joined: 10 Oct 2010

Chris - I haven't gotten to that point yet of doing a multi-page version. That's coming up for me, though, in the next few weeks.

cmontesino
User offline. Last seen 2 days 4 hours ago. Offline
Joined: 21 Oct 2010

I'm having a hard time implementing this along with the Director Class too. Any updates? Thanks

DavidBFox
User offline. Last seen 2 weeks 14 hours ago. Offline
Joined: 10 Oct 2010

@cmontesino, sorry, no updates. I'm wrapping up our first title, which does not use this code. Our second, though, will use the Director Class along with an updated version... so maybe I'll post, but it will be at least a month or so.

cmontesino
User offline. Last seen 2 days 4 hours ago. Offline
Joined: 21 Oct 2010

Ok, it's fine I'll just keep trying :) I'll post any update.

cmontesino
User offline. Last seen 2 days 4 hours ago. Offline
Joined: 21 Oct 2010

So to use this with Director, just follow 7 steps. Search comments beginning with --1 , --2 and so on. I'm sure it's not the best way, but it seems to work. :)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
--------------------------
-- SoundTextSync
--------------------------
--1 Create a group for the text
textGroup = display.newGroup()
 
--2 I made blackText and redText local, don't ask me why
local blackText={}
local redText = {}
local soundLength = 0
 
-- Data is obtained by loading the full audio track into Audacity (freeware
-- sound editing program), selecting each word and creating a label. The word spoken
-- is the name for the label. Then Export Labels to create a text file with the data.
-- Times are in seconds, so multiply by 1000 to convert to milliseconds.
-- Add newline=true where you want a CRLF (so the following line starts on a new line)
local voice = {
        {start=0.359055, out=0.643383, name="WAS"},
        {start=0.643383, out=0.752740, name="the"},
        {start=0.752740, out=1.210217, name="night"},
        {start=1.210217, out=1.600257, name="before"},
        {start=1.600257, out=2.456886, name="Christmas"},
        {start=2.456886, out=2.646438, name="when"},
        {start=2.646438, out=3.018252, name="all"},
        {start=3.018252, out=3.258837, name="through", newline=true},
        {start=3.258837, out=3.384598, name="the"},
        {start=3.384598, out=3.924092, name="house", newline=true},
        {start=4.441715, out=4.576588, name="Not"},
        {start=4.576588, out=4.702349, name="a"},
        {start=4.702349, out=5.088743, name="creature"},
        {start=5.088743, out=5.316570, name="was"},
        {start=5.316570, out=5.892517, name="stirring,"},
        {start=6.016455, out=6.236991, name="not"},
        {start=6.236991, out=6.464818, name="even"},
        {start=6.464818, out=6.555038, name="a"},
        {start=6.555038, out=7.217559, name="mouse;", newline=true},
        {start=7.879168, out=7.964831, name="The"},
        {start=7.964831, out=8.606392, name="stockings"},
        {start=8.606392, out=8.741266, name="were"},
        {start=8.741266, out=9.173226, name="hung"},
        {start=9.173226, out=9.360955, name="by"},
        {start=9.360955, out=9.459376, name="the"},
        {start=9.459376, out=9.935079, name="chimney"},
        {start=9.935079, out=10.164729, name="with"},
        {start=10.164729, out=10.634963, name="care", newline=true},
        {start=11.269234, out=11.469721, name="In"},
        {start=11.469721, out=12.042023, name="hopes"},
        {start=12.042023, out=12.211526, name="that"},
        {start=12.211526, out=13.124657, name="St. Nicholas"},
        {start=13.124657, out=13.543858, name="soon"},
        {start=13.543858, out=13.749814, name="would"},
        {start=13.749814, out=13.875574, name="be"},
        {start=13.875574, out=14.287486, name="there;"},
}
 
-- Using the voice table, display the text by creating an object for each word
local function displayText(params)
        local x,y,color,alpha = params.x, params.y, params.color, params.alpha
        local xOffset = 0
        local words={}
        local fontSize = 30
        local lineHeight = fontSize*1.2
        local space = fontSize/5
        local fontType = "Baskerville"
 
        for i = 1,#voice do
          words[i] = display.newText(voice[i].name, x+xOffset, y, fontType, fontSize)
          --3 Insert words into Director textGroup
          textGroup:insert(words[i])
          words[i]:setTextColor( color[1],color[2],color[3])
          words[i].alpha = alpha
          -- convert to lower case and remove punctuation from name so we can use it 
          -- to grab the correct audio file later
          words[i].name = string.lower(string.gsub(voice[i].name, "['., ]", ""))
          words[i].id = i
          --  calculate the duration of each word
          words[i].dur = (voice[i].out - voice[i].start) * 1000
          if params.addListener then
                        words[i]:addEventListener( "tap", speakWord )
                end
          xOffset = xOffset + words[i].width + space
          if voice[i].newline then y = y + lineHeight; xOffset = 0 end
        end
        local soundLength = voice[#voice].out*1000
        return words
end
 
-- Add a button to start the talking
local function displayButton(params)
        local x,y,w,h,color = params.x, params.y, params.w, params.h, params.color
        local rect = display.newRect(x, y, w, h)
        --4 Insert rect into Director textGroup
        textGroup:insert(rect)
        rect:setFillColor(color[1],color[2],color[3])
        return rect
end
 
-- not currently being called
local function stopNBC()
        media.stopSound("1.mp3")
end
 
-- The button was pressed, so start talking. Highlight each word as it's spoken.
-- trans1 is the delay in milliseconds for the fade to red as the word is spoken
-- trans2 is the delay in milliseconds for the fade back to black after the word is spoken
-- use a shorter trans1 value for snappier response. Longer trans2 to make the fade
-- back to black more fluid.
local function saySentence(event)
        local delay1, delay2, trans1, trans2 = 0,0,200,400
        media.playSound("1.mp3")
 
        --5 Created this NEWsoundLength, it's the same as soundLength made before
        local NEWsoundLength = voice[#voice].out*1000
 
 
        transition.dissolve(blackButton,redButton,trans1,0)
 
        --6 change soundLength to NEWsoundLength.
        transition.dissolve(redButton,blackButton,trans1,NEWsoundLength+trans2)
 
        for i = 1,#voice do
                -- start transition early so it's full red by the time the word is spoken
                delay1 = voice[i].start*1000 - trans1
                if delay1 <0 then delay1 = 0 end
                -- add extra time at the end so we never finish before the fade is complete
                delay2 = voice[i].out*1000 + trans2
                transition.dissolve(blackText[i],redText[i],trans1,delay1)
                transition.dissolve(redText[i],blackText[i],trans2,delay2)
        end
end
 
-- A word was touched, so say it now. Disabled during speech.
function speakWord( event )
        local word = event.target
        local id, name, dur = word.id, word.name, word.dur
        local trans = 200
        -- was the sentence button pushed or this word already active? If so, return now.
        if blackButton.alpha == 0 or word.alpha ~= 1 then return end
        -- make sure the duration is at least longer than 2 transition times
        local dur = dur + 2*trans
        media.playEventSound("snippets/"..name..".mp3")
        transition.dissolve(word,redText[id],trans,0)
        transition.dissolve(redText[id],word,trans,dur)
        return true
end
 
-- initialize
local textx = 297
local texty = 455
 
redButton = displayButton{x=textx - 60,y=texty,w=60,h=60,color={255,0,0}}
blackButton = displayButton{x=textx - 60,y=texty,w=60,h=60,color={60,60,60}}
blackButton:addEventListener( "tap", saySentence )
 
blackText = displayText{x=textx,y=texty,color={60,60,60},alpha=1,addListener=true}
redText = displayText{x=textx,y=texty,color={255,0,0},alpha=0}
 
--7 Insert textGroup into localGroup, or your main group
localGroup:insert(textGroup)
<lua>

DavidBFox
User offline. Last seen 2 weeks 14 hours ago. Offline
Joined: 10 Oct 2010

Nice, cmontesino!

staytoooned
User offline. Last seen 12 weeks 1 day ago. Offline
Joined: 8 Sep 2010

Thanks David for posting this template. I was able to modify it for our project and it works beautifully. Also, coincidentally, I'm a fellow MWA member. I have one question about using your code. I want to come up with an automated way of formatting the table of words for each line:

start=0.165732, out=0.334650, name="I"

in some sort of spreadsheet/database program like Excel to streamline the process. Do you have any recommendations? It's kind of tedious to go into the Audacity text file and add the: start=, out=, name="" for each entry.

Thanks.

darcy@newleaf.com.au
User offline. Last seen 34 weeks 5 days ago. Offline
Joined: 14 Jan 2011

I this this tool might be useful in terms of automation:

http://www.stefanbion.de/cueltool/index_e.htm

staytoooned
User offline. Last seen 12 weeks 1 day ago. Offline
Joined: 8 Sep 2010

Thanks David. That looks great, but it is only compatible with Windows:( I'll see if I can get a hold of a PC.

DavidBFox
User offline. Last seen 2 weeks 14 hours ago. Offline
Joined: 10 Oct 2010

Hi staytoooned - that was darcy4 with the tool suggestion. I went back and took a look at my solution. I had pasted the Audacity output into an Excel file. There I ended up converting the times in seconds to the milliseconds that Corona expects (rather than doing the conversion at runtime), calculating the duration of each word, and adding the other formatting.

You can download the excel file I'm using here:
http://www.ElectricEggplant.com/corona/sound_data_conversion.zip

I paste the data in on the left section, and let the right section do the formatting. Then I copy the right section into my text editor (BBEdit).

You'll have to update your code to use this new format, though. Or modify the Excel file to provide what you want.

Hope this helps!

staytoooned
User offline. Last seen 12 weeks 1 day ago. Offline
Joined: 8 Sep 2010

Thanks David! You template and Excel solution are fantastic! It was very generous of your to contribute this code! Good luck with your eBook series:)

darcy@newleaf.com.au
User offline. Last seen 34 weeks 5 days ago. Offline
Joined: 14 Jan 2011

I also have to say "Thanks!" - we used this code in our kids book for iOS - Busy Bunnies [/itunes.apple.com/us/app/busy-bunnies/id432467432?mt=8] and it worked great.

DavidBFox
User offline. Last seen 2 weeks 14 hours ago. Offline
Joined: 10 Oct 2010

That's great darcy4 and staytooned!!

shbieta
User offline. Last seen 2 years 13 weeks ago. Offline
Joined: 29 Mar 2011

Hi All,
First of thanks for the good work, this module and others are very helpful.
I'm writing a kid book in Arabic (RTL language)
My question is: does Corona SDK support RTL text, I need this to make it work with SoundTextSync because now It draws the text LTR and that make a lot of alignment problems

Thanks

flyingaudio
User offline. Last seen 30 weeks 6 days ago. Offline
Joined: 24 Mar 2011

DavidBFox, nice work! You have a good eye for sytle. Thank you.
On line 29 of your first post, you asked if there is a better way. Since media.playSound has been deprecated, changing to audio.loadSound would solve two issues, the deprecation, and preloading the sound without a need for stopping it immediately again. Of course, line 132 would need to change to audio.play.

ignis075
User offline. Last seen 51 weeks 1 day ago. Offline
Joined: 4 Oct 2010

Great work on the "Director" version of this, cmontesino! I'm tweaking it now to suit my project.

But I'm curious... as I look through the code, I don't see any method to clean up the tap listeners on Director page transition (as Ricardo Rauber has said, Director does not clean up listeners, the user has to program this per-project).

I imagine alot of listeners might clog up an eBook; if you're applying a listener for every word, this is dozens of listeners *per module*.

Am I just over-thinking this? Have any users experienced problems with this?

I'm going to start building a function to clean up all of the listeners on Director page swap... should be simple enough: just loop through the "voice" table and remove the listeners the same way that they were added (via loop).

Brent Sorrentino
Ignis Design

RobMabry
User offline. Last seen 1 year 35 weeks ago. Offline
Joined: 11 Feb 2011

Hey guys. Looking for a little help with this. I've been following Dr. Rafael Hernandez' tutorial on Scene Transition using the Director Class and trying to incorporate this code to use Director.

When I try to follow the inline code instructions in step 7, I am confused. It says to --

--7 Insert textGroup into localGroup, or your main group
localGroup:insert(textGroup)

When I try to use this to insert into my "mainGroup", I get the following error. I also tried to return the localGroup but this doesn't work either.

Runtime error
...ions/CoronaSDK/SampleCode/StoryBookDirector/game.lua:187: attempt to index global 'mainGroup' (a nil value)
stack traceback:
[C]: ?
...ions/CoronaSDK/SampleCode/StoryBookDirector/game.lua:187: in function 'new'
.../CoronaSDK/SampleCode/StoryBookDirector/director.lua:116: in function 'loadScene'
.../CoronaSDK/SampleCode/StoryBookDirector/director.lua:394: in function 'changeScene'
...ions/CoronaSDK/SampleCode/StoryBookDirector/menu.lua:53: in function <...ions/CoronaSDK/SampleCode/StoryBookDirector/menu.lua:51>
?: in function <?:214>

This is my code. Any help sorting this out would be appreciated. Note: I changed the local group name from "textGroup" to "localGroup" in my code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
--[[
Copyright (C) 2011 by Rafael Hernandez(a.k.a cheetomoskeeto)
 
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
 
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
 
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
]]--
 
module(..., package.seeall);
 
function new()
        local localGroup = display.newGroup();
                
--2 I made blackText and redText local, don't ask me why
media.playSound("1.caf")
media.stopSound()
 
local bg = display.newRect( 0, 0, display.contentWidth, display.contentHeight )
bg:setFillColor(255,255,255)
blackText={}
redText = {}
local soundLength = 0
local blackText={}
local redText = {}
local soundLength = 0
 
-- Data is obtained by loading the full audio track into Audacity (freeware
-- sound editing program), selecting each word and creating a label. The word spoken
-- is the name for the label. Then Export Labels to create a text file with the data.
-- Times are in seconds, so multiply by 1000 to convert to milliseconds.
-- Add newline=true where you want a CRLF (so the following line starts on a new line)
local voice = {
        {start=0.359055, out=0.643383, name="WAS"},
        {start=0.643383, out=0.752740, name="the"},
        {start=0.752740, out=1.210217, name="night"},
        {start=1.210217, out=1.600257, name="before"},
        {start=1.600257, out=2.456886, name="Christmas"},
        {start=2.456886, out=2.646438, name="when"},
        {start=2.646438, out=3.018252, name="all"},
        {start=3.018252, out=3.258837, name="through", newline=true},
        {start=3.258837, out=3.384598, name="the"},
        {start=3.384598, out=3.924092, name="house", newline=true},
        {start=4.441715, out=4.576588, name="Not"},
        {start=4.576588, out=4.702349, name="a"},
        {start=4.702349, out=5.088743, name="creature"},
        {start=5.088743, out=5.316570, name="was"},
        {start=5.316570, out=5.892517, name="stirring,"},
        {start=6.016455, out=6.236991, name="not"},
        {start=6.236991, out=6.464818, name="even"},
        {start=6.464818, out=6.555038, name="a"},
        {start=6.555038, out=7.217559, name="mouse;", newline=true},
        {start=7.879168, out=7.964831, name="The"},
        {start=7.964831, out=8.606392, name="stockings"},
        {start=8.606392, out=8.741266, name="were"},
        {start=8.741266, out=9.173226, name="hung"},
        {start=9.173226, out=9.360955, name="by"},
        {start=9.360955, out=9.459376, name="the"},
        {start=9.459376, out=9.935079, name="chimney"},
        {start=9.935079, out=10.164729, name="with"},
        {start=10.164729, out=10.634963, name="care", newline=true},
        {start=11.269234, out=11.469721, name="In"},
        {start=11.469721, out=12.042023, name="hopes"},
        {start=12.042023, out=12.211526, name="that"},
        {start=12.211526, out=13.124657, name="St. Nicholas"},
        {start=13.124657, out=13.543858, name="soon"},
        {start=13.543858, out=13.749814, name="would"},
        {start=13.749814, out=13.875574, name="be"},
        {start=13.875574, out=14.287486, name="there;"},
}
 
-- Using the voice table, display the text by creating an object for each word
local function displayText(params)
        local x,y,color,alpha = params.x, params.y, params.color, params.alpha
        local xOffset = 0
        local words={}
        local fontSize = 30
        local lineHeight = fontSize*1.2
        local space = fontSize/5
        local fontType = "Baskerville"
 
        for i = 1,#voice do
          words[i] = display.newText(voice[i].name, x+xOffset, y, fontType, fontSize)
          --3 Insert words into Director localGroup
          localGroup:insert(words[i])
          words[i]:setTextColor( color[1],color[2],color[3])
          words[i].alpha = alpha
          -- convert to lower case and remove punctuation from name so we can use it 
          -- to grab the correct audio file later
          words[i].name = string.lower(string.gsub(voice[i].name, "['., ]", ""))
          words[i].id = i
          --  calculate the duration of each word
          words[i].dur = (voice[i].out - voice[i].start) * 1000
          if params.addListener then
                        words[i]:addEventListener( "tap", speakWord )
                end
          xOffset = xOffset + words[i].width + space
          if voice[i].newline then y = y + lineHeight; xOffset = 0 end
        end
        local soundLength = voice[#voice].out*1000
        return words
end
 
-- Add a button to start the talking
local function displayButton(params)
        local x,y,w,h,color = params.x, params.y, params.w, params.h, params.color
        local rect = display.newRect(x, y, w, h)
        --4 Insert rect into Director localGroup
        localGroup:insert(rect)
        rect:setFillColor(color[1],color[2],color[3])
        return rect
end
 
-- not currently being called
local function stopNBC()
        media.stopSound("1.mp3")
end
 
-- The button was pressed, so start talking. Highlight each word as it's spoken.
-- trans1 is the delay in milliseconds for the fade to red as the word is spoken
-- trans2 is the delay in milliseconds for the fade back to black after the word is spoken
-- use a shorter trans1 value for snappier response. Longer trans2 to make the fade
-- back to black more fluid.
local function saySentence(event)
        local delay1, delay2, trans1, trans2 = 0,0,200,400
        media.playSound("1.mp3")
 
        --5 Created this NEWsoundLength, it's the same as soundLength made before
        local NEWsoundLength = voice[#voice].out*1000
 
 
        transition.dissolve(blackButton,redButton,trans1,0)
 
        --6 change soundLength to NEWsoundLength.
        transition.dissolve(redButton,blackButton,trans1,NEWsoundLength+trans2)
 
        for i = 1,#voice do
                -- start transition early so it's full red by the time the word is spoken
                delay1 = voice[i].start*1000 - trans1
                if delay1 <0 then delay1 = 0 end
                -- add extra time at the end so we never finish before the fade is complete
                delay2 = voice[i].out*1000 + trans2
                transition.dissolve(blackText[i],redText[i],trans1,delay1)
                transition.dissolve(redText[i],blackText[i],trans2,delay2)
        end
end
 
-- A word was touched, so say it now. Disabled during speech.
function speakWord( event )
        local word = event.target
        local id, name, dur = word.id, word.name, word.dur
        local trans = 200
        -- was the sentence button pushed or this word already active? If so, return now.
        if blackButton.alpha == 0 or word.alpha ~= 1 then return end
        -- make sure the duration is at least longer than 2 transition times
        local dur = dur + 2*trans
        media.playEventSound("snippets/"..name..".mp3")
        transition.dissolve(word,redText[id],trans,0)
        transition.dissolve(redText[id],word,trans,dur)
        return true
end
 
-- initialize
local textx = 297
local texty = 455
 
redButton = displayButton{x=textx - 60,y=texty,w=60,h=60,color={255,0,0}}
blackButton = displayButton{x=textx - 60,y=texty,w=60,h=60,color={60,60,60}}
blackButton:addEventListener( "tap", saySentence )
 
blackText = displayText{x=textx,y=texty,color={60,60,60},alpha=1,addListener=true}
redText = displayText{x=textx,y=texty,color={255,0,0},alpha=0}
 
        mainGroup:insert(localGroup);
        return localGroup;
 
end

mike4
User offline. Last seen 26 weeks 5 days ago. Offline
Joined: 22 Feb 2010

My first guess is that you are inserting localGroup into the mainGroup, and then returning the localGroup.

Try changing 187, and 188 to this:

1
2
mainGroup:insert(localGroup);
return mainGroup;

If that doesn't do it, I will take a closer look.

renvis@technowand
User offline. Last seen 4 hours 39 min ago. Offline
Joined: 22 Jun 2011

I can't find any deceleration mainGroup

when using director you just have to create any local group and put all your display objects in to that group and return it from the new() function.
in your example you have created localGroup so you don't have to use
mainGroup:insert(localGroup);
just return the localGroup.

so u try removing
mainGroup:insert(localGroup);
and see how it works

RobMabry
User offline. Last seen 1 year 35 weeks ago. Offline
Joined: 11 Feb 2011

I tried "return localGroup;" which is what I thought I should be doing based on the tutorials for Director Class and the code suggestions in this thread. It doesn't error but it also doesn't seem to run the the "game" code from the localGroup either.

I'll keep digging. I'm sure it's something mundane and foolish.

Thanks to you both for your willingness to help.

renvis@technowand
User offline. Last seen 4 hours 39 min ago. Offline
Joined: 22 Jun 2011

I cannot find the line which adds bg to localgroup. Make sure that you have inserted all display objects to the local group which is returned as part of director scene.

renvis@technowand
User offline. Last seen 4 hours 39 min ago. Offline
Joined: 22 Jun 2011

:) let me know how it works..

RobMabry
User offline. Last seen 1 year 35 weeks ago. Offline
Joined: 11 Feb 2011

Good catch. Will give it a try a little later today. Thanks for your help.

RobMabry
User offline. Last seen 1 year 35 weeks ago. Offline
Joined: 11 Feb 2011

Okay...lesson for newbies. If you test an app designed for the iPad and use the iphone emulator and that application uses a black background...it's very possible that it might look like there's just a blank black screen. You might think that your code is not working.

You might start asking for help on the forums.

You might be baffled as to why your code doesn't work.

You might decide that all this "coding" stuff is not for you.

You might eventually decide to change the mode to "iPad."

You might then realize that your code was working all the time, you just couldn't see it.

You might want to slit your wrists in frustration...or...you might decide to keep on going.

This would then allow you to endure another 6 hours of frustration until you realize that Lua variables are case-sensitive.

All this and more can be yours.

Thanks to those who offered assistance...problem solved. Looking forward to my next "dumb ass" moment.

mike4
User offline. Last seen 26 weeks 5 days ago. Offline
Joined: 22 Feb 2010

Hey all,
If anyone is still subscribed to this conversation, do me a favor and have a looksee over here:

https://developer.anscamobile.com/forum/2011/08/30/build-works-simulator-not-device

The long short of it, is that I got everything working in the simulator, but the text doesn't show up when I build for device...

If you have any ideas, reply here or there. I am desperate.

corona208
User offline. Last seen 3 weeks 23 hours ago. Offline
Joined: 13 Mar 2012

In the above code of "soundTextSync" are works correctly but one problem is that how to set pause/resume button on glow text?? i can't done it yet! anyone have any idea plz share it

corona208
User offline. Last seen 3 weeks 23 hours ago. Offline
Joined: 13 Mar 2012

Hi David!
I want to get minor help from u that in the above code of "soundTextSync" are works correctly but one problem is that how to set pause/resume button on glow text?? i can't done it yet! anyone have any idea plz share it

DavidBFox
User offline. Last seen 2 weeks 14 hours ago. Offline
Joined: 10 Oct 2010

Hi Corona208 - sorry, I don't have a solution for this. Maybe someone else can chime in?

Tima
User offline. Last seen 36 weeks 6 days ago. Offline
Joined: 18 Jan 2011

Hi Corona208 and DavidBFox,

I've a got a solution for pause and resume function.
You need to changejust anything.
The problem of missing function soud.position() can be replaced with a timer function.
There is just a little problem, but can be fixed with 'approximation'.
this variable: local enSecondes = (event.count/9.25)
Normally, you need to have this: event.count/10 but if you set 10, the word are little bit out sync
If you change this 10 by 9.25 (approximation find after multiple test) it's ok on my simulator and two android device (Htc One X and Acer A500 Tab)
It's not a perfect solution, but the only available i've found at this time for replace sound.position()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
-- play
local function saySentence(event)
        chantMusic = audio.loadSound("10.mp3")
        chantChannel = audio.play( chantMusic, { channel=1, loops=0 }) 
        -- init Timer for replace the missing soud.position()
        monTimerA = timer.performWithDelay(100, monTimer, 0 )
end
-- pause
local function stopNBC()
        audio.pause( chantChannel )
        -- pause timer !
        timer.pause( monTimerA )
end
 
local function replayNBC()
                audio.resume( chantChannel )
                -- timer resume
                timer.resume( monTimerA )
end
 
 
local function monTimer(event)
        local enSecondes = (event.count/9.25) -- 10 dont work!!??
        local enSecondes2 = (event.count/9.45)
 
        for i = 1,#voice do
-- RED word
                if ( enSecondes >= voice[i].start and enSecondes <= voice[i].out ) then
                        blackText[i].alpha=0
                        redText[i].alpha=1
                end
-- BLACK word
                if enSecondes2 > voice[i].out then 
                        blackText[i].alpha=1
                        redText[i].alpha=0
-- cancel timer after the last word. You must specify it in the word table
-- {start=13.875574, out=14.287486, name="there.", theLastWord=true},
                        if voice[i].theLastWord then timer.cancel( event.source ) end
                end
        end     
end

Mudit
User offline. Last seen 20 weeks 5 days ago. Offline
Joined: 17 Jan 2012

Hi DavidBFox and cmontesino,

Thanks for sharing a wonderful solution for sound and text sync.

I am a noob in Corona, still I used it with storyboard api, added your code in external module, tweaked a bit as per my requirements, added a function to destroy text and audio and it worked like CHARM.

Many thanks again.

kilopop
User offline. Last seen 1 day 4 hours ago. Offline
Joined: 24 Jul 2012

Wondering if anyone still subscribing to this post could help to get the text paragraph to be center justified?

I'm thinking the secret lies in line 138

if voice[i].newline then y = y + lineHeight; xOffset = 0 end

Instead of setting xOffset to 0, it could set to a number which would be the difference between the x size of the textGroup and the x size of the new line.

Being a noob, I would would appreciate some pearls of wisdom as to how to actually achieve the text to be center justified.

Ideas?

Mudit
User offline. Last seen 20 weeks 5 days ago. Offline
Joined: 17 Jan 2012

Hi Kilopop,

I achieved it by adding a parameter after setting newline e.g. { newline=true, padding = 30} and in code it is done by checking if newline param is set, check for padding. In case if padding is present, add padding to xOffset:

if voice[i].newline then
y = y + lineHeight;
if voice[i].padding then
xOffset = voice[i].padding;
else
xOffset = 0;
end
end

The code is tweaked little bit as per my needs but hope the concept will help you.

kilopop
User offline. Last seen 1 day 4 hours ago. Offline
Joined: 24 Jul 2012

Hey thanks Mudit, that works really well. Smart thinking.

I got a little lost trying to generate code which could automatically calculate what the padding would be in order to keep lines center justified. Something like (not code but ideas):

(width of group text appears in - length of line up until newline) / 2..?

Thanks again.