Performance - Part 6

May 31, 2009 · by Kevin Runde

Performance Tuning – The end of the beginning

When I started these posts I did not want to end up with a list of “If you have problem X then do Y”. Unfortunately, I have used information like that and its usually been wrong. Instead, I wanted to pass on the the process of how I do performance tuning hoping it will help someone else. The tools, the metrics, the patterns don’t do you any good if you do not know how to apply them. If you don’t understand “you don’t know what you don’t know” then you’ll probably stumble around and may never find the cause of the problem and just point at something like the database telling people it’s the databases fault. If you are a persuasive person you’ll get everyone to believe you and your company may spend millions of dollars on hardware and software to upgrade the database. Congrats you’ve probably just made the project a financial failure. If the database is truly the problem you should be able to go to the database team and show them what specific SQL calls are slow. Maybe the data model is messed up or the application is doing way more commits than any database can handle or maybe the database just needs to have the indexes rebuilt or it’s statistics updated.

Filed in: Team Member Blog Comments (0)

Performance - Part 5

April 30, 2009 · by Kevin Runde

Performance Tuning – Being scientific and being human

You are moving along, making baby steps towards your goal of ultimate speed and efficiency. Hopefully, all your tests and data have made it easier to get everyone on board. I’ve always found it has in the past. Sure, maybe you need to do some more tests with new metrics to make others happy, but that is a good thing?

Filed in: Team Member Blog Comments (0)

Performance - Part 4

March 26, 2009 · by Kevin Runde

Performance Tuning – So why is it slow

So now you have some data (from your tests) and you know what the data should tell you (your goals and other’s expectations). If you are still reading this then your data is probably not where it should be. So, what do you do? How do you succeed? I use a scientific method. First, you take your data and you identifyONE and only ONE problem point. Sit back, think about what may be causing it, but DO NOT CHANGE THE CODE TO FIX IT YET! See if you have data that shows you what is going wrong. Remember, you should have more than one metric to show you exactly where the problem is. If you don’t have enough metrics, then add more. This is where Meta Programming, AOP, and Dependency Injection are very valuable. I want to have the least impact on the code.

So, now you have data that tells you exactly where the problem is. There are probably several things that could be causing the problem. DONOT FIX THEM ALL AT ONCE. Fix one and only one. Test it several times to prove it is fixed. Then, undo that change and prove it is broken again. Then put the fix back in and validate that it still works. Yes, this means you are going to be doing a lot of tests. Welcome to science. To me, this is where programming really is a science. You need to treat it as such, which also means communicating all of your data. Saving all of your data and being able to compare test runs against any other test run. Excel can work wonders here or create a small database app that can read in all of your performance data and analyze it. Have it write out to Excel so that you can generate graphs or a command line graphing tool like GNUPlot. Yes, writing that app will take a bit, but you will be doing so many tests that it will pay out in the long run. It’s not just the amount of time it will save you, either. It will reduce errors and allow you to stay focused on the big issues. During this time, I often find myself climbing into my own little world and only dealing with others when I need something from them. When you realize that you are doing that, stop. Get up and go find someone to talk to for a while. Go read /. or something. When solving big problems like this, you often need to walk away from the problem for a while before your find the answer. If you want to understand this more, I recommend The Pragmatic Programmers book Pragmatic Thinking and Learning: Refactor Your Wetware. It explains why you need to get out of your L Mode. If you don’t know what that means, then read the book.

So, which problem do you try to fix first? The one that is taking up the most amount of time, of course. Oh, sure, you agree with me now, but too many times I have seen people say, “Well I don’t know how to fix that one yet so I’ll go solve these little problems with caching right now. That is a simple change to my Spring Config.” DONOT DO THAT! Once you have fixed your biggest problem the application will behave differently. Those changes you were going to do before may not be needed now. Remember, this is real science and science takes time and does not respect a time line. So, don’t waste your time on changes you may not have to make, or worse, introduce new bugs or performance issues. Remember, performance tuning is like Ogres and each layer can be a vastly different Ogre. Don’t be afraid to do big massive changes to the system. Just don’t do them unless you have a lot of data to back up the decision. You should have designed the system so that you can make changes easily. You also should be able to tell management how confident you are in the change based on the data you have.

Next time, I’ll talk about the side affect of being scientific, and being human.

-Kevin

Filed in: Team Member Blog Comments (0)

Performance - Part 3

February 28, 2009 · by Kevin Runde

Performance Tuning – Your first few tests

So, now you know how to prove what you think you know. You know your goals and what others are expecting. So let’s start testing, right? NOPE! I know, I know. Kevin, when do we actually start testing? How are we going to get any where if we don’t start testing? Well, now that you know where you are going (your goal) and any side trips you have to make (the other expectations) you need to be able to tell that you have accomplished all of these things. In other words, what metrics do you need and how are you going to record them? So, for each goal and expectation, what metrics are you going to use? Yes, I said metrics. You can’t assume one metric will be enough. After all, you don’t know what you don’t know so you need other metrics to validate the metric you are using to be able to check off each goal and expectation.

For example, if you are creating a web based application, you are probably using a testing tool to generate various user loads to see how the system performs. How do you know that tool is the right one? Well, if you are using Apache you can set up the logs to report response time. (NOTE: The Apache response time includes sending the data to the user.) So, if the tool does not agree with the Apache logs, you have an issue. Is the tool downloading other content, like images, CSS, and JavaScript, that you are not seeing in your Apache logs? Does rendering the page in the browser take a long time? Maybe the page is too complex with lots of tables and such or maybe your tool is having problems. Maybe it is doing too many requests per server. What is the CPU, network, disk, and memory usage stats on the testing boxes?

Notice something? That one metric of page response time has suddenly turned into several more metrics. This is why we are not testing yet. You need to look at each metric and figure out how you will validate it and, when something is wrong, how you will break it down even further. Going back to the web page example; if page requests are taking over 5 seconds, what is really taking the time? If the tools, metrics and the Apache logs are close then the problem isn’t at this level and you need to dig down deeper. If Apache hands off the page request to an application sever (like what often happens in Java Web Apps) can you tell how long the application server is taking. Remember, I said the Apache response time included the time to transfer the data. Maybe you have a network bottleneck. So in your application code, log out how long you think it takes you to process a page request. Don’t reinvent the wheel here either. There are often great tools you can use likeJAMon for Java. It even has a Servlet Filter you can use to monitor your web application that will generate stats. But wait, Kevin. I have a nice Network usage chart that shows there isn’t a problem. Does that chart show every switch and router in between the tool and the test server? Probably not. Again, you don’t know what you don’t know so find ways to prove what you think you know. Don’t assume anything. Getting paranoid yet? No. No one told me to say that. If you got that, then maybe, just maybe, you are starting to realize you don’t know what you don’t know.

Now, you don’t need to implement each metric right now. Make sure you have ways to record each major metric. Don’t waste time putting in additional metrics now. Remember, performance tuning is like an Ogre. As you peel back each layer, you will be adding more metrics. For now just think about what other metrics you might use and find out how to get more. Look at what additional metrics each tool,API, and component can give you. If possible, get people to design how they would add the other metrics.

Now run some tests. Run the same test 3 or 4 times. Don’t make any changes! Just run the tests. Then, look at the results. Are your results the same across all the runs. If they aren’t, don’t worry. It just means you don’t know what you don’t know. Take the metrics that are changing and come up with new metrics to validate those metrics and drill down into what is really going on.

Finally, make sure you know how much your metrics cost you. Any time you observe something (record data), you affect it. If you run your code in a profiler, it runs significantly slower. It will make out-of-process calls look faster because they are not running in the profiler. If you have to, run a test with and with out your metrics using a wall clock to check if the tests take roughly the same amount of time. If they don’t, you need to compensate for that.

Next time, I’ll talk about how to run the tests and, once you’ve found a problem, what do you do.

-Kevin Runde

Filed in: Team Member Blog Comments (0)

Performance - Part 2

January 20, 2009 · by Kevin Runde

Performance Tuning - And So It Begins

So, you have just been told you are responsible for performance tuning the application that will save the company. Congrats and welcome to your worst nightmare! :-) Okay, it won't really be that bad, because you won't have time to sleep, so you won't have nightmares. Thanks to JMS for the title to this post. In case you ever happen across this JMS, thank you for creating Babylon 5, and if you ever make Mr. Garibaldi fall of the wagon again... sorry, I am getting off topic.

The first thing you should do is find out who decides what fast enough is. I mean who really decides it. Sure, the users may say they can control if a project goes live or not, but who really does. Usually some management group. Then, you need to find out what is fast enough to those people. Is 90% of responses in 5 seconds fast enough? Probably not for an Internet site, but might be for an Intranet site that does a lot of back end work and has a lot of data on each page. If 5 seconds is acceptable, then don't waste your time trying to speed up requests to 1 second. Once every page is responding at 5 seconds, then make things better. Also, if the goal they give you seems impossible, don't say so yet. Remember, you don't know what you don't know, so how can you make an argument that it is impossible?

Make sure these people receive constant communication from you. Flood them with it if needed. Make sure they understand the metrics you will be reporting to them. That way they can see the progress. You are probably not going to be able to go in and make a few changes, and then say "done". So, they will need to understand how to see that little successes are being made. Don't be afraid to tell them you are trying something and then report that it didn't work. Performance tuning is all about failure and learning from it. Performance tuning is like an Ogre. It has layers! Sorry, had to make the Shrek reference. It's one of the tricks I use to remember that concept.

Okay, great, you've got your goal. STOP! Don't start running performance tests yet. Who are the rest of the people that influence if you are successful or not? Is there a user that must be pleased? If so, are that user's expectations reasonable and in-line with your goal? Do they have any additional metrics you should be reporting? If not, how will you handle that? Are there admins who run the servers that are watching how much of their systems you are taking up? If you take up too many resources, they are going to have to request more hardware and software. So, what are their concerns and what sort of price tags are attached to them? Do their expectations match up to your goal? Do they have any additional metrics you need to report?

which means other expectations Odds are that every person's expectations will not match the goal. So, you need to start deciding how will you handle this. Performance tuning is a balancing act. You can always do more, but when is enough enough, and how do you get everyone to go along with that? There is no answer I can give you for this. It depends on your situation and the culture you are in. If the project really is the project to save the company, then management probably wants it out now,are irrelevant to management. If that is the case, watch out. You are going to be target for a great many things during the tuning and after. If no one uses the product, and it doesn't save the company, it may be because the users were right and the system wasn't fast enough, even though management said it would be effective.

I suppose JMS would sum up this whole article with: "Whom do you serve? What do they want?"

-Kevin Runde

Filed in: Team Member Blog Comments (0)

Performance - Part 1

December 22, 2008 · by Kevin Runde

Performance Tuning

For years now, I have ended up doing performance tuning. Well, I really call it performance triage. The first one was a Java Web Crawler that was a beautiful OO creation. I mean this looked wonderful on paper and in code. We even had an independent analyst come in and examine the entire project. He looked through the code and the db for over a week. His final report came out and all he could say is we might have problems with too many method invocations. Later, we turned the system on and it blazed across the Internet at 0.2 pages per second. HUH? Yeah, 0.2 pages per second. Eventually we were able to tune the crawler and got it going well over 100 pages per second. Yep, 0.2 to 100 pages per second. Oh, and just in case you are wondering, method invocations were never a performance problem. ;-) Since then, I have been on other projects and the same sort of things happen over and over. Big companies, small companies, medium sized companies, big project, small projects, critical projects, and unimportant projects. They all were the same and they all were solvable. Over the next several blogs I will tell you my tips and tricks for performance tuning.

To start with, though, you need to understand something. This is probably the most important TRUTH I know.

“You don’t know what you don’t know”

Yeah, I know what you are probably thinking. This guy is an idiot. Of course I don’t know what I don’t know.BUT do you really realize it? Truly learning this lesson is a very humbling experience. You have to question every “fact” you use in your thought process and ask yourself how do I know this. What proof do I have? How reliable is this proof? For example, one data point is meaningless; it is just a dot on a graph. I know, “but Kevin, two data points is a line.” So, most people do a test twice and draw their line. I say: “But three tells you if that line is really a curve!” So, the more times you can reproduce a result, the more reliable the “fact” is. Also, in order for anything to even be a “fact,” I say it needs to be reproduced at least 3 times. Also, who did the test? What recorded metric data? Was the test done in the same way and with the same data as last time? What about the code? Is it exactly the same and if so, how do YOU know? Do you know if the network and performance testing systems can really handle the load you are putting on them? Are you sure there isn’t a switch some place that only allows 10 Mbps, but you are trying to generate over 45 Mbps? Are both those speeds Mega Bits Per Second or Mega Bytes Per Second? How many objects are the JSP pages really creating? Does our response time include the time to transfer the data to the client or just how long it takes you to generate the content to send to the user? How often is the Garbage Collector running and for how long? What are the CPU loads on every machine involved (the servers running the application, the database servers, the server recording the metrics, the server running the tests, and each and every box used to play the test)? How much can you really trust your team members? If you don’t know the answers to these questions, then you don’t know what you don’t know. The next posts are going to be about learning what you don’t know. Because once you know what you don’t know, you can come up with a game plan to find the problem. Once you know the problem, it is trivial to create solutions. Yes, as in multiple solutions. Implementing them may not be so easy, but that’s not part of performance tuning.

-Kevin

Filed in: Team Member Blog Comments (0)

NEW SITE AND NEW BLOGGING SOFTWARE

November 13, 2007 · by Kevin Runde

NEW SITE AND NEW BLOGGING SOFTWARE

As I’m sure you noticed we now have a new web site at http://www.QWANtify.com. This post is mainly to test the new blogging software. If you see this then that means I can now write up my messages on Google Docs and then post to my QWANtify blog. AWESOME!!

Filed in: Team Member Blog Comments (0)