How to change the computer name on a server configured by Puppet

Question

I'm dealing with a very large list of large data frames (~2GB). To save space and reduce file size, I want to remove some elements of the list that are all NA. As part of the operation I need to gather and then bind into a single data.frame.

Here's an example:

library(tidyr)
library(dplyr)

a <- data.frame(x=rep(1,3), y1=1:3, y2=1:3)
b <- data.frame(x=rep(2,3), y1=NA,  y2=NA)
c <- data.frame(x=rep(3,3), y1=1:3, y2=NA)

l <- list(a,b,c)

t <- lapply(l, function(x){
  gather(x, key="type", value="value", -x) # %>%
    #remove list element here %>%
    #do other operations like mutate here
}) %>%
  bind_rows

The result of this includes some data.frames that are all NA for my values of y.

I would like to remove elements from the list completely. If remove all rows with NA it still leaves me with an empty list element, which then crashes further calculations with mutate or other operations.

I'm trying to take care of this operation with the first call to lapply because I find that doing filtering after that requires a lot of memory (often crashing after maxing out the 16GB I have on this computer). In the title when I say "list" I'm referring to this apply statement.

In this example the result should look like:

> t[-(7:12),]
   x type value
1  1   y1     1
2  1   y1     2
3  1   y1     3
4  1   y2     1
5  1   y2     2
6  1   y2     3
13 3   y1     1
14 3   y1     2
15 3   y1     3
16 3   y2    NA
17 3   y2    NA
18 3   y2    NA

Solution 1

EC2 will start with a unique name if the AMI that's starting it has been configured to set the Computer Name to the ip- in the EC2 Config Service.

See this: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html

Solution 2

For new relic, you just need to change your hostname as that is the identifier they use. Hostname changes do not require any restart, not at least for Linux. You can do following in puppet exec:

hostname `ifconfig eth0|grep "inet addr"| awk -F ":" '{print $2}'|awk '{print $1}'`

This will set your hostname as IP address and newrelic can use this as identifier. It would be easier for you too to diagnose issues as they will be shown against IP address of the machine.

Comments

[SHYAMLAL]

Yeah, I thought about simply adding an exec for changing the machine name on the node when it's being configured by puppet, but unfortunately changing the machine name in windows requires a system reboot and I'm trying to prevent rebooting my systems just to install new relic.

[SHYAMLAL]

thanks for the answer, I guess I should update my question to specify that I am trying to do this both for Linux nodes and Windows Server 2008 R2 nodes.

[Lloyd Christmas]

I'm trying to avoid create separate variable, i.e. doing as much as I can with pipes. If I create separate variables I run out of memory. Can I bring the subset statement inside the lapply statement and use pipes?

[Yannis Vassiliadis]

@LloydChristmas with the example you provided, you can! In general, I don't see why you wouldn't, given that the output from subset is a list itself.

[Lloyd Christmas]

Thanks. This does seem to work. Although it results in a nested lapply statement. I will test this on my larger dataset and see if it is computationally viable. Revised code: lapply(l, function(x){ gather(x, key="type", value="value", -x) %>% subset(unlist(lapply(l, function(x) !all(is.na(x[,-1])))))}) %>% bind_rows

[Yannis Vassiliadis]

With the revised code you provided at don't get the same results. I get the right results, if I first use gather to get t and then subset it.

[Lloyd Christmas]

You're right. This greatly simplified code works well. If you update answer to reflect it I will accept. lapply(l, function(x){ gather(x, key="type", value="value", -x) %>% subset(!sum(!is.na(value)) == 0) }) %>% bind_rows