Automated HTML Format E-Book Concordance Generator

Started a new programming project; an automatic concordance generator for HTML based E-Books. Given a list of terms, it searches an HTML document and inserts local hyperlinks where it finds each term. Once it is done, it returns a list of all the local hyperlinks as another HTML document to be used as the starting point for a hand edited index. Created to speed up process of creating an index for an E-Book, a free .PDF based alternative program was found but not one for .HTML formats.

It works as intended at it's most basic functionality, but needs more refinement in the design. For example when two concurrent paragraphs contain the same term the program will find and mark both paragraphs, this is an unintended result. Index entries do not need multiple links to the same page from the same index entry as it will bloat the index and confuse users.

Noted Issues and areas for improvement:

• Unnecessary index entries from close proximity terms.
• No method of specifying areas to skip concordance generation.
• Duplicate search terms with different punctuation get treated as different terms, leading to duplicated results.
• Performance speed is terrible because of the recursive nature of the current search algorithm. I noted a 5 minute average for my 150 page test document, stripping the header and title HTML (which leaves body of content untouched) resulted in 01:40 minute average, a 33.3% improvement. I am confident the performance speed can be improved further.
• Still have not implemented multi-threading in my applications, leading them to appear unresponsive while working on an operation for long periods.
• Has no functionality for opening documents with preexisting index and being able to edit, add, or merge with existing results.
• Ideally would have a graphic user interface where editors could quickly browse the document and manually add in index entries, as well as those generated by the concordance, and be able to dynamically manage the data.
• Ideally would like to have dictionary support, being able to categorize terms as nouns, verbs, singular or plural etc, and then being able to compare them against words of similar meaning and make smarter recognitions.

In short trying to have more levels of smarter automation, to save editors time and effort.

A pipe dream would be to create an artificial intelligence capable of doing more than concordance, recognizing ideas conveyed in text and then being able to create a 100% machine generated index, rather than concordance. The massive raw data needed for word identification and networking, and pattern recognition algorithm would be huge endeavors in themselves, but not impossible. Perhaps worth some pursuit when time and money are in more abundance.

VGRemix Entry Finalized

Not happy with how it turned out, but I ran out of time; I started late and had to finish early, so it's good for what it is I guess. Development Thread on Polycount also check it out in 3D on Sketchfab!





Should've given the boat and tank double their current texture size, they are just way too pixelated when scaled with the rest of the scene elements.
Lack of vibrant colour, was going to do explosion effects with those iconic swirls, but again just ran out of time.
Some grass on small cliff shelfs around the tower would've helped I think as well.
And getting proper rigging for the characters to I could've posed them more dynamically. 


Ah well live and learn, it was a fun experience!
I got a lot of assets now, enough to get the ball rolling on a game perhaps?

Entered the Polycount VGremix Contest.

Yep, so I figured I needed to get back to being productive and that competing in the contest would be good for me to help build up momentum and practice.

 My entry is a planned depiction of the fictionalized Normandy Beach Landings in WW2 ( Think Saving Private Ryan, or Medal of Honor, or Call of Duty for video game sources ) remixed with the Legend of Zelda The Wind Waker art style. I'm probably biting off more than I can chew, but it's good to challenge yourself - and plus I should have a pretty good set of art assets to use for further use. 

I was honestly surprised that I didn't see any other entry using a military shooter as one of their bases, but perhaps people are tired of it, and other games lend themselves better to colourful stylized interpretations.

Probably the easiest bit of work so far, still have to do the environment textures, characters, vehicles and special effects. 7 more days to go.

Dezoomify Downloader Progress

Well for the last 5 days or so I've been diligently working on the de zoomify downloading program. Now it is finally working with all the features that I planned it to have : batch downloading, user specified file formats, user specified download folder and renaming mask, and most recently : the ability to find zoomify folder links on any given webpage.

However it doesn't have an updating progress bar, or any sort of visual indicator of download progress because the application is not multithreaded.  Just starting to read the documentation on threading is disorienting, let alone implementing it. I would really like to implement threading so that users who aren't running the program as a debug build in visual studio can see the progress of the download and what it is currently doing. But as it stands now, I was extremely busy before starting the project and I don't know if I have the time or resources to continue developing a feature I have no idea how long it will take to implement.

For now I'm calling it done, having served the purpose I created it for - although it's not in the best of shape. Anyways the source code is freely available : https://github.com/RedRogueXIII/DeZoomify_NET I would like to eventually return to it and add threading, but for now I don't know.

Anyways, at the moment I'm looking for a way to batch compress lossless quality pngs, since I now have 2 GBs of high res images that could be much smaller.

De-Zoomify Image Downloader Origins

As for the story behind why I started this program in the first place; I'll give you some background, I'm an addict for collecting image references for rare guns. One of my hobbies is to create 3D models of guns for use in video games, so it's good to have references on hand for when I want to start a new project.

A 3D model I made of an M1A1 Thompson, a submachine gun used by allied forces in WW2

Cowan's Auction house is currently selling the Richard L. Wray Collection, a collection of old, extremely rare, antique, machine guns in excellent high resolution and beautiful photography. Trend being that after the auction is done, these images will disappear into the void means I couldn't just bookmark it and come back later. So I tried to download them, one by one, finding and figuring out how to use the original web based implementations of Lovasoa's Dezoomify.

Over two hours I got the images for four lots. There is one hundred lots in the auction. So my options are to either take forever to do it manually or actually use my brain and put together something that will save me tons of time now and any other time I would need it later. I'm also on the lookout for a programming job, so having experience and a useful program to show definitely helps.

Anyways for the TL;DR : OM NOM NOM NOM MACHINE GUN REFS.

De-Zoomify Image Downloader

For the last three days I've been writing a program to download images that are used in the web service Zoomify. Zoomify splits high resolution images into tiles and only loads them as needed, so it's pain to get the whole high resolution image by downloading the tiles manually.

The program, which I'm calling DeZoomify Downloader is being written in C# using Visual Express and also up on Github: https://github.com/RedRogueXIII/DeZoomify_NET . I've done a smaller .NET application before, but trying to go all out on this one, with preference saving, batch handling, file format options, and a lot of fancy usability options.

At the moment it probably isn't too stable - I know I haven't been sanitizing or checking my inputs, so it's not extensively bug tested. Oh well it's still not close to having all the features I want it to have, but at least it's basic functionality works at the moment. ( Major credit goes to Lovasoa and his simpler web based version : https://gist.github.com/lovasoa/770310  who's work helped served as an understanding of how to get the images in the first place.) So still plenty of time to go back and fix and potential crap outs in can have.

Just today I got batch download working, so there are still two major features I would like to add still: finding all zoomify image links on a given webpage, and multithreading the application so the damn progress bar updates while it's downloading and it doesn't do the "program is not responding, uh oh better kill the app" business while it's busy downloading megabytes of information.

I don't know if it's worth going into detail with any of the mechanics of the program so far, seeing as how the dezoomify process isn't something I came up with, and how almost everything else is handled by the extensive .NET libraries.  Also the entire source code is free to view, download, modify, and run from Github. Perhaps the process of identifying and pulling of the zoomify links from webpages that are displayed to users may be a nice topic for a blog post. That or a horror story of me trying to learn multithreading with 50 tabs open of different "First Must Read Articles About Multithreading".

Geez, so much headache just to make an updating progress bar.

I was entertaining the idea of a separate blog just for non game development stuff, but eh I wouldn't post frequently in either one so all it goes in one blog.

Break in Posts; EDU-Game Updates

Well there was another long break in posts, but unusually that was caused by me being too busy to post anything. Quite a nice change from having a long break with very little done.

As the educational game stands, we have the demo level done - all functionality, just a few tweaks and customizations of Unity's poor GUI and it'll be ready to go. Now we can build levels relatively quickly, so the only impediments are actually coming up with the level designs, and making any outstanding art assets ( 3 environments & some misc models).

Aside from that though, I've been putting together my resume. Got to start looking for a stable job, because as fun as game development is, it pays poorly compared to the time invested in it.

Video Update

Hey finally have a neat, err boring video to show off the game's progress so far. Well it should be nice to gather my thoughts/ self-critique on where to improve.

Some things right off the bat:
- Spelling, grammar errors and in general bad dialogue.
- Tutorial message popups would be nice.
- Have unique asset backgrounds when using unique systems (phone, newspaper, computer, etc)
- effects code is actually a big mess that I need to reorganize
- player/location management is also a mess that needs to be thought out and reorganized
- Awkward victory/end screen, needs something better
- Items are usable from outside the intended perspective (only when zoomed in should they be usable - I could try encapsulating them in a collider that zooms in when clicked, that might work.)
- Interactable objects need some sort of highlighting when mouse over them in the world.
- Clicking on buttons sometimes activates the button and the item underneath it, this shouldn't happen
- I have a event system complete with it's own mission and objective system for parallel mission progress, instead of the singular linear mission system currently used. Would be nice to switch it up.
- Work on more generic and reusable triggers and locking behaviors.
- Test/ Integrate have a plan for eventual localization of all in game text.
- Build a Esc/ pause menu
- Build an options menu
- Ensure that the game's menus work on different resolutions.
- Saving and loading level progress and achievement.
- World is bland and dull. 
- The fact that the current environment still needs more polish, but there are still 3 other environments to be done.
- Lighting is fullblown in places and needs to be redone and optimized.
- No sound effects or any audio stimulus aside from repetitive background music.

I'm very sure the list of shit suxs - improve it, can and does go on. At the moment I can't think of any more to elaborate on, but if anyone is actually reading this I'd love to hear anything about what you hate or don't like about this.

New Year

Well it's been pretty quiet the last few months, but progress has picked up in the last few weeks. We have a functional demo, although there are numerous graphical bugs that I am currently working on.

I've been writing my own systems documentation - something I should have done months ago. It serves as a good way to communicate how my systems are intended to be used, and forces me to sit down and think hard about my design and execution choices.

With the all the gameplay systems set and the necessary code and functionality ready, designing additional levels for the complete game shouldn't take that long. However there are still three additional 3D environments incomplete, so those will be big time sinks. Oh well, I've always found making art to be more enjoyable than programming.

As for right today, I made a high quality phone asset that will likely be used often:

Creating it's own drawing coordinates and GUI system may be troublesome, but I don't actually know if it'll be a requirement. We could probably get away with leaving it as a static background item while just using the regular GUI commands to make the illusion of icons on the phone's screen.

Let's try a more industrious method to enviroment creation

Right-o, so for a long while I've been on programming duties, but yesterday after finding out I had coded duplicate functionality for something else my partner had done (lead designer also). I decided that 1: I have no idea what's going on in the design, 2: I enjoy doing art better than programming, 3: whatever man, there are still 3 environments to go (which I hope doesn't change on me later). So yep, now I'm going to work on two of the three outstanding environments (the apartment could use some more love too) which are a gym and a park. I had set a self-imposed deadline of having level blockouts done one per day so that I would have a start for both by Friday, but level design is one of the many things I under estimate.

Whatever, enough words, here's my progress so far:

To be a locker room

To be a transition area between the locker room and the main areas

To be a gymnasium, setup with basketball nets and a suspended indoor track

To be a workout room overlooking the pool

To be a pool

To be the entrance to the environment with a snack area

Coming up with floorplans are suprisingly a pain in the ass for just arranging blocks. I tried to plan for lines of sight - so that there is always something interesting to see and explore, and to have very short travel times when walking between main areas. I've tried to design the floorplan for the gym a few times before and always got sidetracked placing areas that the player would never go to ( like offices) or having unacceptably long and winding corridors between areas.

As you can see yes I have blockouts of the level, no they're not any good at describing what the final product should be, so I think I may need another day working on these images. What I'll likely do is use my poor painting skills to paint over these screenshots and see how that works out. Once these poor excuses of concept art are done and approved, I'll be doing the same for the park - maybe even the hospital if they turn out to be not terrible.

Experimentation with procedural levels

Well while doing the loading bar, I found out some neat commands; Application.LoadLevelAdditive namely. And in a case of "when you have a hammer, everything looks like a nail" decided to change how levels were setup. Since the game only takes place in 4 environments and has much more levels I thought it would be good to build the levels procedurally by reusing the environments by additive loading them into a blank scene that contains the relevant level data.

It works but on experimentation it leaves this slow blank awkward pausing screens that I rather not deal with. So scrapping that idea, I just dumped all the level data into the start menu scene, deactivated and ready to go on level select. It's not ideal but it works. The issues with this method is that it is harder to control scene specific objects with a script that isn't even based in the same scene. - Say like I needed to script some doors locked, I probably would have to create a level specific reference list script in the scene just to get easy access to variables needed to properly setup the environment for the current level.

Also got the generic event system going and it seems to work without problems for purpose of objective tracking, haven't extensively tested it or seen how it handles large amounts of data, but handle problems as they come along I guess.

Generic Event System

Well continuing on from last post I got that resolution independent GUI layout helper function done. It took the whole day because I basically over thought the math necessary to scale elements down between a minimum and maximum. Figuring out an equation for linear interpolation of 2D elements, or scaling by the difference in area and then applying Mathf.Clamp on the results? I went with the latter, much easier to code and understand.

So onto the event system which is supposed to be my silver bullet for solving all my woes in tracking objectives and attachments progress. It's under 40 lines of code.  - Well uh maybe I learned from the GUI code kept it simple, stupid. All it is, is a glorified list with time stamps, so yes the size seems right. But like everything else all it's dirty hidden problems will only show themselves through testing and implementation. Now for the fun part of hooking it up and giving it real feeds and having objectives query it.

Also on the to do list is now loading bars/ screens, and run time asset loading. I've got loading screens working now thanks to a quick Google search and the ease of use that is sometimes Unity. Just need to work on making nice progress bars, but I'm sure a search will turn up something.

As for run time asset loading, well I'm not looking forward to that. Yes there are numerous benefits to the technique, like patching without having to recompile or redistribute the whole folder, but then I also need to worry about asset management and file type parsing. For a first game, it doesn't strike me as a priority over getting shit working and actually enticing to play.

It always looks like there is no end in sight, but just gotta get what I can get done.

Got a working prototype finally

Well it's been awhile, I honestly don't even want to keep track of how many days it has been, but last Friday I managed to scrape together a crude, but working version of the game - main menu, levels, objectives, actions, and recognizing victory. Of course since it was all done in a rush and with lots of hard coded quick fixes it is an absolute mess and I have to go back and reorganize and improve everything, but at least it's a step forward.

WIP Start menu, the bars are animated.

As for today I'm going to make a tool for converting pixel coordinates into resolution independent percentages that should make creating a GUI layout for multiple resolutions a lot easier and automated.

What it should take into account is the screen resolution the layout is currently designed for, where each element is and what point it appears to anchored to and then generating the coordinates for any resolution that it was not designed for. I don't think this will be that hard, so today I may also be able to get started on a generic event system.

The generic event system would be a log of every action recorded in one place that would allow for classes to tell each other what they are doing without explicitly hard coding in conditionals into their functions. This is useful for checking for achievements and objectives, which as of now are hard coded into the relevant behaviors. Anyways more on it as I go.

Day 130: Actions

Well today was pretty good, got the action sequence part working and it's ok. Still a lot of issues need more work, but hey at least it works.

The code for moving the avatar to a specific spot, orienting correctly, playing an animation for a specified time and then moving back to it's original position, all while allowing for it to be skipped at any time. This is all in one script, and I think it's a right mess. I used so many boolean state flags to keep track of it all that I'm definitely sure that there is a more simple and elegant way to handle all this.

I have a vague idea for a sequencer structure, that basically takes in function delegates which would be single parts of the big script, executing them, waiting for next and going on until all queued operations are finished. Pretty much a hack job at a cutscene controller or game scripting from SDKs.

Still have to tie all the various components that could make up an action together; animation, menu, and camera stuff all need to be properly sequenced and setup for each action - which I haven't gotten around to yet.

I'm sure there is always a better way to do things, but I won't find out until after stumbling in the dark figuring out my own poor implementation to get an understanding of what ever it is that I'm doing.

Day 129: Actions speak louder than words

Well it's been two days since the last post. I kind of got distracted what with Borderlands 2 at it just came out. No, I don't have it, neither am I playing it, but I have been watching videos of someone play through it. More and more I've just been watching people play video games rather than do it myself. Partly because I'm poor and can't afford new releases - both in time and money. Saves time playing the thing as you can skip cutscenes, sidequests and all the other fluff and see how the core game works, what kind of levels it has, and the scale of the gameplay. I've watched a playthrough of Sleeping Dogs in it's entirety, it seems nice with an impressive large representation of Hong Kong, but in terms of gameplay and variety there isn't that much to do. For a game it's size it really seems like the developers could done more with the world and given the player more to do and actual involvement in the story instead of the preset story line. But anyways all that is only somewhat relevant to game dev, being able to watch "Let's Play" videos is all the more common and can serve as a hands off demonstration of the game. Being in the backseat should make it easier to make observations and analysis.

Ah well if I can focus, I still have to do the action controller. Nevermind all the huge distractions from just about everything on the internet.

So how it'll work is that from the main HUD,(1) clicking the action button will cause the avatar to walk to the action zone and start a trigger. (2) This trigger will enable a game object loaded with whatever scripted behaviours need be - (3) something like play a specific animation, add the time past by the action and then shut down the game object and return to the main HUD.

All of this really should be super simple, I've already got 1 and 2 done, just need to do all the custom stuff for 3.

Yeah I really got to get my act together, it's not even the games it's just the internet itself is so detrimental to sustained focus. For the simplest stuff it's like: "Yeah shouldn't take long, let me just read this single page comic strip" and it just spirals out of control from there.

Despite my poor progress still have to maintain a good sleeping schedule - staying up to finish shit is acceptable once in a while but get too comfortable doing it and you'll always wake up tired, out of energy, and just unable to work efficiently and effectively.

Day 127: Pathfinding Differences

Not a lot today, I'm resolving myself to limiting my game playing to only the weekends. So when they come around I'll probably splurge all my time with that. Anyways the quickie right now is changing how path finding works and is setup in our scenes.

It sounds like a big thing, but thanks to an excellent middleware solution, the A* Pathfinding Project by Aron Granberg, it's a pretty simple and painless switch. Of course it's only for one scene, but it only takes  5-10 minutes to setup per scene.

The old path finding sampling grid

The new user placed node system

So yeah, just changed from a sampling grid to user placed nodes that auto link by distance. The nodes render in the game at the moment, but it's an easy fix to just mass select them since they're all under one parent, and then disable or remove the mesh renderer component. Probably disable, since it is useful to have a quick visualization when needed.

As for tomorrow, when I commit really to working on this I'll be working on a sequencer for the various actions that the character can do in the game.  It shouldn't take long, but programming rarely goes as expected for me.

Day 125 : Well that was doozy...

Well you've been seeing all the props I was making for the apartment, but I never posted the apartment scene here. It's actually been awhile since I've worked on this as the last time I did any major art assets for it was in July. August was eaten by another project and then a failure of motivation and effort on my part. I meant to post these screens right after making that last post, but my family dragged me out for a vacation for the week.

The kitchen, empty, uninteresting, and setting the standard for what's to come.

The kitchen, got a ton of empty space to fill in and make interesting. Also there is something wrong going on with those cabinet doors.

Dining room

The dining room, and again a common reoccurring theme is that the rooms will look sterile and empty. I have to work on some minor props to litter the scene and really make the apartment feel lived in.

Living space and floating main character

The main living area along with the main character, which was not done by me. It is probably the most flushed out room so far, but there are pathing issues to work out and some extra props wouldn't hurt. I'm also not that satisfied with the ceiling, as those indentations have some odd lighting artifacts.

The foyer, the way out of this mess.

The foyer, it could do with some actual footwear / a coat hanger / welcome mats and rugs / a side table to help make it seem less lifeless. I also could do a different style door to indicate that it is going out of the apartment.

Living space view 2

Here is the living room again, with a closeup of the potted plant. I was pretty proud of myself when I made that, but now I'm not so sure it's any good. Either way I'll get lots of practice doing plants and vegetation soon - one of the next environments is an outdoor park.

The balcony, what are the neighbors up to?

The view from the balcony. I need to get some deck chairs and figure out something for the sliding doors when the avatar walks in and out of the room. I also need to get a skybox together, as the big grey emptiness has to go. The game also has a time of day system, it's not real time, but still that means that I'll have to figure something out for controlling the skybox and the ambient and direct sunlight.

A box in the sky

Overview of the balcony. I've got other apartments duplicated to modular build the apartment facade with the insides taken out for performance. The camera should never really focus on them so thankfully I don't think I'll need any new props to help differentiate them, but they are there so any oddities with camera movement don't show awkward clipping.

The view to sea, with nothing to see.

A first person view out to sea. Yeah we really need a skybox here.

Bedroom

Yeah I'm no interior designer. The wallpaper is pretty ugly and the tiling can be noticeable. Also missing curtains or blinds or anything to drown out sunlight.  Maybe some small props like a digital clock and a storage box at the base of the bed would help. Getting a digital clock to mirror the in game time would also be great, just that my experience with 3D text in Unity hasn't been the best. (It doesn't backface clip and renders on top of geometry so you'll always see it from the wrong angles.)

Bedroom view 2

The other side of the room, looking into the main washroom.  An issue that will likely be raised in the future is when the characters go to sleep. We have the animations for it, but I'll need to either replace or deform the bed sheets to fit around the sleeping characters if we show them.

The main washroom

The mirrors are another annoyance, at first I found a script that generated real time reflections so that they functioned as actual mirrors, but it was rather buggy since I had two mirrors in the scene for the two washrooms. They also took 50% of the computing power when running the game and lagged it down significantly. Needless to say I cut those and replaced them with a simple cubemap reflective shader for now. I'd like to come up with a better solution, but there are other issues to work at first.

The secondary washroom

Broadway's paintover critique

The mirror manages to perform worse. Counter also looks terrible with the nasty seams. A helpful chap named Broadway over on polycount helped me out with a paintover critique of the room. Well the difference in current state and what it could be is obvious. Still a ton of work to do, and then still there are two more unique environments to do. Hopefully I will be able to reuse a bunch of assets.

The study

The book cabinets where meant to be much more unique and interesting with different colours and text to help build a diverse library. Each stack is meant to be rearranged and have singular books clip in to help build z-axis diversity. But I rushed it and never went back to give it the proper attention I had planned for. Again the scene looks pretty barren.

The iPear, master tool and time waster

Part of the game will have the player sitting down at the desk to answer questions and buy things. The design calls for the camera to shift into first person view while doing so. Here's the view as it is now. ... Well I tried to do low resolution and low poly props, but it's extremely obvious, blurry, and awful here in particular. Iterative design time it is then, as right now these are a must fix.

The hall to the study and washroom

A picture, a decorative table with a flower pot? Anyways this probably needs something, but I'm no interior designer.

Apartment Overview

And that is it for the tour right now. Still a ton of renovations and prop building/fixing/editing to do. Back in July I'd liked to have called it done, but as you can see it's not living up to the potential of what it could be.

Anyways this'll conclude this return to blog posting, hopefully I'll get daily updates going again. I'd like to fix up the apartment starting now, but at the moment there are programming issues to get to first.

Intermission - Took a side project

So yeah I haven't updated this in about 2 months. But I can at least account for one month. Starting late July I was contacted by some friends who were working on a sequel to the iOS shooting game and needed some people to do some gun models. Well a paying job that asked me to do what I enjoy to do on my free time? Sounds very good to me.

I was working on a contract basis, with fixed payment per asset delivered. With the time I spent on each asset I wasn't making a good return on my time, earning probably below minimum wage here in Canada when I took into consideration the hours it took to build each asset. But up until now, I had been making  game ready 3D models of guns for free giving away the models I made for anyone to use. It felt good to step it up from a hobby, and have a commitment to maintaining a high standard of quality in the assets delivered. It was a good opportunity to study my own workflow, and let me know that I still have lots of room to improve.

Anyways the game isn't out yet, but when it is I'll be sure to post back here a link to it, and pictures of all the assets that I did for it. So that took up most of my August, and I didn't get as much done on the educational game as I'd like, but I'm back on that now.

Day 55: More Props

More props done. Now to move onto textures, plus some other tweaks. Then the room textures.

Day 51: Prop Building

Making props, everything is normal mapped, baked from a high polygon model onto a low polygon mesh and is in the Unity game engine. Most are missing textures though, so I still have to make those as well as create more props.
To make:
potted plants, sliding cabinet doors, bed, wardrobe, curtains, sink, mirror, bathtub, kitchen counter and sink,
kitchen islet, windows, balcony door, guard rail, office desk, office chair, pictures, computer, dome light, chadelier, light switch, patio chair, door, shoe rack


Then to texture all the untextured stuff & room textures (floors, walls, ceilings).


Position everything & tweak room dimensions, then bake lighting and hopefully it doesn't look bad.

All this by Friday, go, go, go, go!