Adding 48-bit support for pix2pix

I’ve been experimenting with pix2pix and I’m literally loving it. This is such a great tool to create various image to image mappings.

I’ve currently made some modifications to get it to support 48-bit PNG support. This allows to represent 65536 values instead of the usual 256 values allowing for a higher resolution of data to be stored in an image. The network doesn’t really care because it all gets converted to float values anyway.

Hope this helps someone. I’ve forked the repo at: https://github.com/tharindu-mathew/pytorch-CycleGAN-and-pix2pix

environment.yml for vid2vid for easier setup

Vid2vid seems to be a promising technique for video synthesis using GANs, as of 2019, which is similar in spirit to it’s image counterpart, pix2pix. When setting this up, I found there are some additional requirements needed, than what’s stated at: https://github.com/NVIDIA/vid2vid.

I created my own environment.yml to make this setup easier (pytorch is specifically for CUDA 9.1, CUDNN 7.1.2, but you can edit the yml file and swap out this version based on the available pytorch versions, and it should work).

The gist is available at: https://gist.github.com/tharindu-mathew/7f79433d92d884fa662f6ccf023537e1

How to improve your shadows – Understanding the light projection matrix

When using shadow mapping, the resolution of your shadow buffer is lower than your default color buffer resulting in a low resolution shadow. But, this effect can be somewhat mitigated by programatically computing your light projection matrix so it covers the minimal volume possible.

In the above images, the buffer of the shadow resolution is the same. But, the bounds use for the computation of the light matrix (light_projection_matrix * light_view_matrix) differs. For the high res shadows on the right, the light projection matrix is controlled by the bounds of the view frustrum. In terms of a directional light, this will be an orthographic matrix, and the bounds of the light be the bounds of the viewing frustrum (which can be perspective or orthographic). This makes the resolution of the shadows far more high-res as only a smaller portion of your scene is considered. So, if you choose a reasonable resolution and use a bounds-aware light projection matrix you get good use resolution of your shadows. This can vary for level of detail that you shoot for based on the zoom of your camera.

Binding to framebuffer 0 may cause a blank screen


Today, I encountered an interesting issue that took away a few hours of my (not so) precious life, to debug and understand. I was implementing some basic shadow mapping, which requires you to create a new render target (meaning you need to render to a separate buffer other than the screen). So, we have to switch back and forth between framebuffers. Most OpenGL tutorials out there will simply ask you to bind back to the default framebuffer 0.

Here’s a code excerpt from learnopengl.com site (as of 11/16/2018) from their article on shadow mapping (I love this site, and this in no way a criticism of their content, just using it to point to a probably bug). https://learnopengl.com/Advanced-Lighting/Shadows/Shadow-Mapping

// 1. first render to depth map
glViewport(0, 0, SHADOW_WIDTH, SHADOW_HEIGHT);
glBindFramebuffer(GL_FRAMEBUFFER, depthMapFBO);
glClear(GL_DEPTH_BUFFER_BIT);
ConfigureShaderAndMatrices();
RenderScene();
glBindFramebuffer(GL_FRAMEBUFFER, 0);
// 2. then render scene as normal with shadow mapping (using depth map)
glViewport(0, 0, SCR_WIDTH, SCR_HEIGHT);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
ConfigureShaderAndMatrices();
glBindTexture(GL_TEXTURE_2D, depthMap);
RenderScene();

Binding back to framebuffer 0, as mentioned here, simply output a blank screen for me. And I went through a period of commenting out all my rendering code and enabling line by line (costing me half a day or more) to see what was going wrong. I found the culprit in the line: glBindFramebuffer(GL_FRAMEBUFFER, 0);. Then I suddenly got an idea and executed these lines to find my actual default framebuffer (after disabling any shadow mapping code and simply doing a simple setup for single pass rendering):


glGetIntegerv(GL_DRAW_FRAMEBUFFER_BINDING, &default_draw_fbo_);
glGetIntegerv(GL_READ_FRAMEBUFFER_BINDING, &default_read_fbo_);

And to my surprise, the answer was 3. With allocating a new framebuffer for shadow mapping, this went up to 4 (weird!). I’m using Qt 5.11 as my application framework, and I’m not sure whether this is a bug/feature or what (maybe they use framebuffer 0 to render their own stuff). But, it seems that the default framebuffer cannot be assumed to be 0.

So, if you’re experiencing a blank screen when trying to a render anything that causes you to switch between framebuffers, make sure you find out what exactly you’re default framebuffer ID is. Then, just switch back to this known number, and all will be well.


// also GL_FRAMEBUFFER is deprecated now, simply use
glBindFramebuffer(GL_DRAW_FRAMEBUFFER, default_draw_fbo_);

 

HTH.

The feedback cycle and runtime governance

Introduction

Runtime governance can be defined as the process which allows you to control and manage parameters in your runtime execution environment. A runtime execution environment can vary from a single web server that hosts a simple web page, to gigantic deployments that can span to 1000+ servers. This means the complexity of implementing runtime governance can depend heavily on how complex the runtime environment actually is. A feedback cycle allows to continuously get feedback from the runtime system to govern it more effectively. This article briefly explains how a feedback cycle is important to the runtime governance process.

The feedback cycle

The feedback cycle defines a model that is common to any runtime execution environment. The four stages can be related to any environment regardless of its size.

feedback_cycle-general

Each stage is elaborated on below:

1. Gather data

The gathering of data is the starting point of the feedback cycle. Data can be distributed among many points in a runtime environment. Let’s consider a deployment consisting of a web server. If it is a clustered deployments, all web servers can be potential data collection points. The other option is, if there is a load balancer in front of the cluster, to use the LB as the data collection point. But, this might impact the performance of the load balancer. So, the performance cost needs to be compensated for in terms of additional LBs, depending on the requests per second is affected.

LB web servers

The second option to ponder is what type of data to collect. Typically, the more data you collect the better. This might vary from CPU cycles consumed by the servers to the HTTP headers of all requests. All types of data can be used to generate some sort of useful information.

2. Slice and Dice

After gathering data, the second part of the cycle is to generate useful information through slicing and dicing the data. Real time analysis maybe needed to prevent imminent security threats. For example, a 30 second window maybe enough to send enough requests from multiple IPs to overwhelm a medium sized website.  Batch-based analytics maybe needed for trend analysis over timespans. A combination of batch based and batch based analytics seems to be the most viable option for a variety of requirements of generating useful information quickly and over a large time period. There are various tools in the landscapes of complex event processing and data analytics that allow to rapidly perform analytics to produce useful information.

3. Evaluate Information

The third step of the cycle deals with the fact that each piece of information can present some vital insight about your runtime execution. You notice that website visits are doubling each month. At this point it should be evaluated whether this is just a temporary trend or is this the effect of any recent improvements. Also, you may notice a trend of some increasing downtime among your servers. Maybe, this is related to some sort of attack taking place or some unreliable hardware. Usually,  domain knowledge and solution architecture expertise needs to be heavily utilized to make these insights as these may lead to heavy resource investments.

4. Adjust parameters

The final step of the cycle bridges the feedback cycle to runtime governance. Tuning your parameters to effectively govern your environment is done in this step. This may mean you need more server capacity or some additional steps to boost security and strengthen the site’s resilience based on this observation. Policies can be altered, introduced or decommissioned based on the information given by the feedback cycle.

Conclusion

Based on information gathered from the feedback cycle, the effectiveness of the runtime environment can be judged and altered. It even allows you to get an idea about missing components that are needed for more effective runtime governance. Feedback cycles can also be stretched beyond typical runtime governance applications to understand various trends about server uptime, API analytics and business activity monitoring to get more insight about a business and associated trends.