07 Sep 2016
The TalkAbroad platform is currently experiencing intermittent recording errors that affect our students. The purpose of this post is to explain what is happening, what we’ve done to fix the problem, and to keep you updated on any developments.
This problem is currently affecting approximately 5% of TalkAbroad conversations. In affected conversations the conversation will successfully take place. Unfortunately, the recording will start correctly and then drop one of the audio streams resulting in a recording with only one person. This happens in conversations with poor connections, as the server identifies the poor connection and thinks the user has left the conversation. It then drops the stream from the affected user.
The solution has been identified by our provider and they expect a successful fix to be implemented by Wed, Sept. 14th.
As mentioned, this impacts a small number of students. During this time we will be offering free refunds to any students experiencing this problem. Additionally, students can cancel their conversations at any time and reschedule for after Sept. 14th. If there is any other way we can help facilitate during this period please reach out and let us know.
We will keep this post updated as new information is available.
UPDATE 5: At 8:00AM PST on Thursday, Sep 15th our provider alerted us that 100% of servers have been updated and the issue has been completely resolved. Thank you for your patience in this.
UPDATE 4: Our provider had a few false starts where they found bugs in deploying and had to roll it back. This has caused a delay. They believe they have it all fixed and have implemented the fix on 40% of their servers. The remaining 60% should be rolled out tonight and tomorrow morning (Wed night/Thu morning). We will update this as soon as we hear the final solution has been deployed.
UPDATE 3: The release has run into a problem and the estimation for final deployment has been pushed to Wednesday, Sept. 13th at mid-day.
UPDATE 2: The test and release team finished running tests on Friday and a small portion of traffic was run through patched servers over the weekend. Full release of patch should take place for TalkAbroad servers by the end of the day Monday or Tuesday, PST.
UPDATE: The most frequent quesion we have received is if this error impacts the live conversation for students. The answer is NO. This has no impact on students completing their conversation successfully. They will still be able to have a perfect audio/visual experience with the conversation partners as normal. This error only impacts the resulting audio recording.
On Wednesday, August 24th we reported an isolated incident regarding a student recording that started correctly and midway through dropped the voice of the conversation partner.
On August 31st we were alerted of additional recordings with this error. We reached back out to our provider seeking a resolution.
September 1st, the bug was located by our provider and the ticket was promoted to level P1 which indicates it takes precedent over all other work. A solution was identified and work was begun on fixing the bug.
On September 2nd a workaround was suggested by our provider that would allow our students to complete their conversation without experiencing the errors. At TalkAbroad we worked from Sept. 2-4th to implement the workaround, successfully deploying the fix on Sept. 4th. Errors occurring the night of Sept. 4th and Sept. 5th proved the workaround was unsuccessful and the only solution available needed to be implemented on the side of the provider.
On Sept. 3rd our provider announced that the bug has been fixed but not deployed for clients. Internal testing has begun, and the fix will be rolled out to a small number of servers to assure that it does not introduce other bugs into their system. They estimate full deplyment by Sept. 14th.