Localization (and Unity Importer?) question

Thu 14. Feb 2019, 13:30

Greetings.

I will have to work on a project which asks to be published multiple times during the production process. To get a better grasp of the idea, imagine some sort of game, with lots of text and dialogues (and I mean lots), being divided into some sort of chapters (actually, not really, but that's for the explanation, that will do the trick). Every time a chapter is finished, the project is published as is. So each time, the game gets bigger, since it has the exact same content as before, plus the last finished chapter. So far, so good? Okay, so we continue. Now after each chapter is finished, some revisions on former chapters can occur (like removing or correcting errors, or even re-writing some parts differently), depending on the customers' reviews. The whole project fits in a single Articy Draft project.

Now comes the localization time. Every new update needs to be translated in a few languages. That means that after a new chapter is finished, every modification in former chapters has to be reflected in every languages, and the new chapter has to be translated from scratch. Any text that did not change must not be replaced or translated again. And that's where it becomes tricky.

When you have a full finished project, and you publish it, it's perfectly fine to translate out of an xml, json, or anything along those lines. But when you re-generate that xml or json every few times with modifications to some of the already existing text, plus some new text, how are you supposed to do? Let's say you are at chapter 3 in your development process. You generate the file, and translate everything, fine. Now you are at chapter 4, and you changed a few things in chapter 2. You generate the new file. How can you update the localized data in order to just re-translate what changed, and translate what's new, without having to bother the already translated content?

Being able to write multiple languages directly in Articy Draft would have done the trick, but it's not possible as far as I know. Is there a way I didn't think of that would allow to update the project with some new text (and modifications), and to just have to work on what changed in the localized versions of the files? Since those localized files have to be generated by duplicating-renaming the original file, I really don't see a way, yet.

As you can tell by the title, Unity will be part of the process, I don't know if it can help find an answer to be aware of that.

Any idea?

Maybe some script that compares the former version of the original xml with the updated one, and takes every unique id that corresponds to something that already existed and that didn't change, then using yet another script that use this id list in order to replace in the new localized file every line with those ideas with the former version of the same localized file? I'm sure it would do the thing, but I'm not competent in writing such a script (unless you can lead me to some lessons or tutorials which would get me there in no time). Or maybe is there a tool somewhere that would do just that already (even paid, as long as it's cheaper than the money I can put in it without selling my kidneys)?

Thank you and best regards!
Ylliana
 
Posts: 3
Joined: Thu 3. Jan 2019, 18:31

Re: Localization (and Unity Importer?) question

Thu 14. Feb 2019, 14:07

I thought a bit about my own question and suggestion of an asnwer, and came out with that pseudo code, so far:

XML Compare
Code: Select all
Updated_Original_XML = new XML with up to date original content
Old_Original_XML = earlier version of the original content

For every unique id in Old_Original_XML,
   check if identical id can be found in Updated_Original_XML                   //to avoid storing id that no longer exists
      If so, compare the localizable content of both id
         If identical, store the unique id in a Unchanged_ID_XML list


XML Replace
Code: Select all
Unchanged_ID_XML = list of unchanged id content, according to original XML
Original_Localized_XML = duplicate of the Updated original XML, ready to be translated
Old_Localized_XML = precedent version of the localized file, with already localized text

For every id in Unchanged_ID_XML, find the corresponding id in Old_Localized_XML then,
   store the localizable content bound to that id in variables,
      find the same id in Updated_Localized_XML, and
         replace the localizable content found in it by that of the variables.


Sure that it works. But it has to be coded.
Ylliana
 
Posts: 3
Joined: Thu 3. Jan 2019, 18:31

Re: Localization (and Unity Importer?) question

Fri 15. Feb 2019, 11:13

Hi Ylliana,

first of all, as you might expect, we don't have an out of the box solution to this problem. The localization feature in the unity importer was always meant as a basic feature.
But i might be able to give you some directions how to come up with a custom solution that might work for you, if you don't shy away from a bit of C# coding.
Your pseudo code is actually very close of how i would do it:

First we need to grab the excel once its written from articy (or yourself when dealing with the other languages). To do that, we do the same the unity importer does by writing a custom AssetPostprocessor. This will install a script into unity that is called once a file was saved into your unity project. So we write it to act upon everytime a "xlsx" file is written. Its implemented as an editor script and therefore should be put into an Editor/ folder).
A basic scaffold could look like this:

Code: Select all
class MyLocalizationProcessor : AssetPostprocessor
{
   public static string LocaFileExtension = ".xlsx";

   private static void OnPostprocessAllAssets(string[] aImportedAssets, string[] aDeletedAssets, string[] aMovedAssets, string[] aMovedFromAssetPaths)
   {
      // moved or imported should trigger our script
      List<string[]> changedAssetsArrays = new List<string[]> { aImportedAssets, aMovedAssets };

      // check all places
      foreach(string[] changedAssetsArray in changedAssetsArrays)
      {
         // and all files in those places
         foreach (var asset in changedAssetsArray)
         {
            // get the file extension
            var extension = System.IO.Path.GetExtension(asset);
            
            // make sure the file is actually an excel file
            if (extension != null && extension.ToLowerInvariant().Equals(LocaFileExtension))
            {
               var name = System.IO.Path.GetFileName(asset);
               // excel files generate temp files when they are opened with Microsoft Excel
               // unfortunately they have the same extension, but they are useless to us, so we ignore them
               if (name != null && !name.StartsWith("~$"))
               {
                  // articy writes loca files with the prefix loc_, so we look for it.
                  if (name.StartsWith("loc_"))
                  {
                     // the full path to the saved loca file
                     string pathToArticyLocaFile = System.IO.Path.GetDirectoryName(asset);
                     
                     // TODO: now we could parse the file
                  }
               }
            }
         }
      }
   }
}


The next part would be how to open and read an excel file, this can be extremely difficult or rather simple depending on your needs. I would strongly encourage you to use ClosedXML as it is imho the easiest way to read/write excel files and should be more than enough for your use case. Make sure to put the closedxml dlls into the editor folder. Otherwise your game would have a dependency to it as well.

Next would be to open the written loca file using closedxml, something like this

Code: Select all
Dictionary<string, string> existingEntries = new Dictionary<string, string>()
using (var excelWorkbook = new XLWorkbook(pathToArticyLocaFile))
{
   var rows = worksheet.RangeUsed().RowsUsed().Skip(1); // Skip header row
   foreach (var row in rows)
   {
      string key = row.Cell(1).Value;
      string value = row.Cell(2).Value;
      string isFinal = row.Cell(3).Value;

      existingEntries[key] = value;

      // TODO: work with the values
   }
}


Those should all be necessary tools to get started working on the algorithm.

  • Write all key/value pairs from the loc_ into a C# Dictionary<string, string>
  • Open the "last-send-to-localization" excel and also put it into another dictionary (Excel file doesn't exist when doing it for the first time)
  • Compare both dictionaries and write a new "need-to-be-localized" excel with entries that differ or do not exist. (Ignore this file in the AssetPostprocessor when you write it this way, see below)

This creates the "need-to-be-localized" excel that you can give to a localization office or your translators. Once you get the file back with the changed entries i would also let unity and our scripts handle the update.
Update the AssetProcessor to also read any changed "need-to-be-localized" excels (cleverly skipping when its written by the algorithm) and update the "last-send-to-localization" excels. Using a column inside the excel to keep track of the status would be preferable, similar to the "is final" column. This should make sure that next time a "need-to-be-localized" it only contains the entries that actually where localized.

Now we have our localized texts and just need to write the loc_ excels with the new localized values. This works fine for all additional languages that articy doesn't care about and can be done using similar code we already have. But if you also send the default one (probably "en") to localization/editorial we need to update the inital loc_ as well, but this is the file that articy always writes when exporting. Obviously it would be best to update the texts in articy directly, but this involves writing a MDK plugin. Unfortunately i'm not sure how safe it would be to do the updating just in unity writing the loc_x_en.xlsx as it will trigger another asset import. Worst case scenario it will result in an endless loop. Best case scenario it will create unecessary import runs (First import triggered when articy writes the loc excel unknowingly with old values, second one because your script tries to update the old entries with new ones).
I have no real solution for this problem, but you might be able to work around it.

So there you have it. I know it sounds like a lot of work, especially if you don't want/be able to write the code. But it only sounds intimidating, it should be manageable when broken down.

I hope this helps you and let me know if you need anything else

Best regards

Nico
Nico Probst
Senior Software Engineer | Articy | LinkedIn
User avatar
[Articy] Nico Probst
Articy Staff
Articy Staff
 
Posts: 217
Joined: Wed 23. Nov 2011, 09:45
Location: Bochum

Re: Localization (and Unity Importer?) question

Tue 5. Mar 2019, 09:27

First of all, sorry for the long delay before I reply to you, and a huge thank you for your answer.

I'm currently working on my own implementation of a solution that suits my case, amongst other things I have to work on. My first intent was to finish my thing, and then come here to thank you, but as it takes me longer than expected (mainly due to the fact I have to much work on the side to focus properly on this part yet), I figured I'd come here first, and then finish my work.
Your answer gave me tips and ideas. I decided to not actually do the work inside of Unity for the xlsx management, and went the Python route, outside of Unity, with the openpyxl library. It wouldn't have been too much of a hassle for me to work with C#, as I use it in Unity for the project itself (lately, I delved into binary files managing, which feels a bit more difficult than what you described in your answer, at least to me). But in the end, I wanted to get this part of the process outside of Unity.

So, actually I'm creating multiple python script for different actions and outcomes. One script filters about everything in the xlsx in order to provide a new sheet with only the text to translate, no variable names, no fragments that are not intended to be translated, and so on. An other script takes the translated sheet, and integrates everything that has been translated, while noticing if anything is missing. I have some other scripts and tools in the bake, too.

So far, I have a working script that extract the "ToTranslate" data, and another one doing what my first idead was already. I just have to give it the time needed to write all the script versions I need. That's really all there is to it. No headache, just a bit of time.

By the way, I just learned python by doing this. That was way more easy than I thought. For the record, I have some working knowledge of a few other languages. I'm not a genius or whatever, so it proves Python is really easy. But I have to say I do not agree with some choices made while creating it. Relying on indentation to determine what is inside a method or a conditional, or using the "elif" keyword seems arguable to me. But that's just me and my maniac (Manfred) habits.

Anyway, your answer helped me tremendously, even though I'm not doing it in C# inside of Unity. Parts of the sample code you provided are almost the same in my python scripts!

I'll be back (with some more good news, of course)!

Thank you very much. :)

Best regards.
Ylliana
 
Posts: 3
Joined: Thu 3. Jan 2019, 18:31

Return to articy:draft Unity Importer

Who is online

Users browsing this forum: No registered users and 17 guests

Who We Are
Contact Us
Social Links